Multi-Source Capture

2024-09-20

Feature Overview

Multi-Source Capture is used to manage the audio and video source configurations of each channel. Through this capability, you can flexibly and quickly implement real-time interaction of different audio and video content, such as screen sharing, audio mixing, etc.

The main capabilities and limitations of Multi-Source Capture are as follows:

The publishing channel supports setting multiple audio and video sources, but the same audio and video source can only be occupied by one channel.
The main channel's audio and video sources support dynamic switching, and also support audio mixing capabilities. The auxiliary channel only supports specifying the capture source before publishing stream, and does not support audio mixing.
When capturing screen, only iOS and Android platforms support capturing both video and audio at the same time; other platforms only support capturing video. If you need to capture audio, please implement the relevant logic yourself.

Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Others/src/main/java/com/example/others/multivideosource" directory.

Prerequisites

Before using the Multi-Source Capture feature, please ensure:

You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
You have integrated the ZEGO Express SDK in your project and implemented basic audio and video streaming functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Implementation Process

Warning

Real-time voice products only support setting audio capture sources.

1 Create ZegoExpressEngine engine

For the process of creating a ZegoExpressEngine engine, please refer to Create Engine for implementing video call.

// Create a ZegoExpress instance and listen for common events
    void createEngine() {
        // Create an engine, access the general scenario, and register self as the eventHandler callback
        // If you don't need to register a callback, the eventHandler parameter can be passed null, and you can call the "setEventHandler:" method later to set the callback
        ZegoEngineProfile profile = new ZegoEngineProfile();
        profile.appID = appID;  // Please obtain through official website registration, format: 1234567890L
        profile.appSign = appSign; // Please obtain through official website registration, format: @"0123456789012345678901234567890123456789012345678901234567890123" (64 characters in total)
        profile.scenario = ZegoScenario.BROADCAST;  // Specify the use of the live streaming scenario (please fill in the scenario suitable for your business according to the actual situation)
        profile.application = getApplication();
        engine = ZegoExpressEngine.createEngine(profile, null);
    }

// Create a ZegoExpress instance and listen for common events
    void createEngine() {
        // Create an engine, access the general scenario, and register self as the eventHandler callback
        // If you don't need to register a callback, the eventHandler parameter can be passed null, and you can call the "setEventHandler:" method later to set the callback
        ZegoEngineProfile profile = new ZegoEngineProfile();
        profile.appID = appID;  // Please obtain through official website registration, format: 1234567890L
        profile.appSign = appSign; // Please obtain through official website registration, format: @"0123456789012345678901234567890123456789012345678901234567890123" (64 characters in total)
        profile.scenario = ZegoScenario.BROADCAST;  // Specify the use of the live streaming scenario (please fill in the scenario suitable for your business according to the actual situation)
        profile.application = getApplication();
        engine = ZegoExpressEngine.createEngine(profile, null);
    }

2 Set audio and video capture sources

Call the setVideoSource interface to set the video capture source.

/** Use camera as video capture source */
engine.setVideoSource(ZegoVideoSourceType.CAMERA);

/** Use camera as video capture source */
engine.setVideoSource(ZegoVideoSourceType.CAMERA);

Notes

If you need to use custom video capture as the video capture source, you need to set the video capture source type to ZegoVideoSourceTypeCustom, call the enableCustomVideoCapture interface to set the video frame data type to be sent, and call the sendCustomVideoCaptureRawData, sendCustomVideoCaptureTextureData, or sendCustomVideoCaptureEncodedData interface to send video frame data to the SDK. For details, please refer to Advanced Features - Custom Video Capture.
If you need to use a media player as the video capture source, you need to call the createMediaPlayer interface to create a media player first, and then call the setVideoSource interface to set the video capture source type to ZegoVideoSourceTypePlayer and set the media player instance index used by the video capture source, and then use the newly created media player to load resources and play. For details, please refer to Media Player.
If you need to use screen sharing as the video capture source, you need to set the video capture source type to ZegoVideoSourceTypeScreenCapture and enable screen sharing to push screen sharing video data to the SDK. For details, please refer to Screen Sharing.

Call the setAudioSource interface to set the audio capture source.

Set audio capture source only

/** Use microphone as audio capture source */
engine.setAudioSource(ZegoAudioSourceType.MICROPHONE);

/** Use microphone as audio capture source */
engine.setAudioSource(ZegoAudioSourceType.MICROPHONE);

Set audio capture source and set audio mixing configuration at the same time

ZegoAudioSourceMixConfig audioSourceMixConfig = new ZegoAudioSourceMixConfig();

/** Mix the sound played by the media player with sequence number 0*/
audioSourceMixConfig.mediaPlayerIndexList = new int[1];
audioSourceMixConfig.mediaPlayerIndexList[0] = 0;

/** Mix the sound played by the audio effect player with sequence number 0*/
audioSourceMixConfig.audioEffectPlayerIndexList = new int[1];
audioSourceMixConfig.audioEffectPlayerIndexList[0] = 0;

/** Use microphone as audio capture source and set audio mixing configuration */
engine.setAudioSource(ZegoAudioSourceType.MICROPHONE, audioSourceMixConfig);

ZegoAudioSourceMixConfig audioSourceMixConfig = new ZegoAudioSourceMixConfig();

/** Mix the sound played by the media player with sequence number 0*/
audioSourceMixConfig.mediaPlayerIndexList = new int[1];
audioSourceMixConfig.mediaPlayerIndexList[0] = 0;

/** Mix the sound played by the audio effect player with sequence number 0*/
audioSourceMixConfig.audioEffectPlayerIndexList = new int[1];
audioSourceMixConfig.audioEffectPlayerIndexList[0] = 0;

/** Use microphone as audio capture source and set audio mixing configuration */
engine.setAudioSource(ZegoAudioSourceType.MICROPHONE, audioSourceMixConfig);

Notes

If you need to use custom audio capture as the audio capture source, you need to set the audio capture source type to ZegoAudioSourceTypeCustom, and call interfaces such as sendCustomAudioCaptureAACData or sendCustomAudioCapturePCMData to send audio data to the SDK. For details, please refer to Advanced Features - Custom Audio Capture and Rendering.
If you need to use a media player as the audio capture source, you need to set the audio capture source type to ZegoAudioSourceTypeMediaPlayer, call the createMediaPlayer interface to create a media player, and then use the newly created media player to load resources and play. For details, please refer to Media Player.

For the process of logging in to Room and publishing stream, please refer to Login Room and Publish Stream for implementing video call.

4 (Optional) Switch audio and video capture sources

Call the setVideoSource interface to switch the video capture source.

/** Use screen sharing as video capture source */
engine.setVideoSource(ZegoVideoSourceType.SCREEN_CAPTURE);

/** Use screen sharing as video capture source */
engine.setVideoSource(ZegoVideoSourceType.SCREEN_CAPTURE);

Call the setAudioSource interface to switch the audio capture source.

/** Use media player as audio capture source */
engine.setAudioSource(ZegoAudioSourceType.MEDIA_PLAYER);

/** Use media player as audio capture source */
engine.setAudioSource(ZegoAudioSourceType.MEDIA_PLAYER);

5 Stop publishing stream

For the process of stopping publishing stream, please refer to Stop Publishing and Playing Stream for implementing video call.

/** Stop publishing stream */
engine.stopPublishingStream();

/** Stop publishing stream */
engine.stopPublishingStream();

FAQ

How to capture screen video and system audio?

If you need to capture screen video and system audio, please refer to Screen Sharing.

Does it support setting multiple media players?

Yes, but note that the same player instance can only be occupied by one channel.

What are the limitations when combining video sources and audio sources?

The limitations for combining video sources and audio sources are as follows:

The main channel cannot use the media player's video or audio source.
When the auxiliary channel uses a media player as the audio and video source, the main channel needs to use and start the microphone device (enableMicrophone). If you do not need microphone audio on the main channel, you can disable audio publishing through mutePublishStreamAudio while keeping the microphone enabled.
Only when the main channel uses the camera video source or microphone audio source can the auxiliary channel copying the main channel video source or audio source take effect.
The main channel's audio source cannot be set to None, otherwise the audio cannot be rendered.