Multi-Source Capture

2024-09-20

Feature Overview

Multi-source capture is used to manage audio/video source configurations for each channel. Through this capability, you can flexibly and quickly implement real-time interaction with different audio/video content, such as screen sharing, audio mixing, etc.

The main capabilities and limitations of multi-source capture are as follows:

The publishing channel supports setting multiple audio/video sources, but the same audio/video source can only be occupied by one channel.
The main channel's audio/video sources support dynamic switching and audio mixing capability. The auxiliary channel only supports specifying capture sources before publishing and does not support mixing.
When capturing screen, only iOS and Android platforms support simultaneous video and audio capture; other platforms only support video capture. If audio capture is needed, developers need to implement related logic themselves.

Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please check files in the "/ZegoExpressExample/Examples/Others/MultiVideoSource" directory.

Prerequisites

Before using the multi-source capture feature, ensure that:

You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, see Console - Project Information.
You have integrated the ZEGO Express SDK into your project and implemented basic audio/video streaming functionality. For details, see Quick Start - Integration and Quick Start - Implementation Flow.

Implementation Flow

Caution

Real-time audio products only support setting audio capture sources.

1 Create ZegoExpressEngine Engine

For the process of creating the ZegoExpressEngine engine, please refer to Create Engine in implementing video call.

ZegoEngineProfile *profile = [[ZegoEngineProfile alloc] init];
profile.appID = [KeyCenter appID];
profile.appSign = [KeyCenter appSign];
profile.scenario = ZegoScenarioDefault;
[ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];

ZegoEngineProfile *profile = [[ZegoEngineProfile alloc] init];
profile.appID = [KeyCenter appID];
profile.appSign = [KeyCenter appSign];
profile.scenario = ZegoScenarioDefault;
[ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];

2 Set Audio/Video Capture Sources

Call the setVideoSource interface to set the video capture source.

/** Use camera as video capture source */
[[ZegoExpressEngine sharedEngine] setVideoSource:ZegoVideoSourceTypeCamera];

/** Use camera as video capture source */
[[ZegoExpressEngine sharedEngine] setVideoSource:ZegoVideoSourceTypeCamera];

Note

If you need to use custom video capture as the video capture source, you need to set the video capture source type to ZegoVideoSourceTypeCustom, call the enableCustomVideoCapture interface to set the video frame data type to be sent, and call sendCustomVideoCapturePixelBuffer, sendCustomVideoCaptureTextureData, or sendCustomVideoCaptureEncodedData interfaces to send video frame data to the SDK. For details, please refer to Advanced Features - Custom Video Capture.
If you need to use a media player as the video capture source, you need to call the createMediaPlayer interface to create a media player first, then call the setVideoSource interface to set the video capture source type to ZegoVideoSourceTypePlayer and set the media player instance index used by the video capture source, and then use the newly created media player to load resources and play. For details, please refer to Media Player.
If you need to use screen sharing as the video capture source, you need to set the video capture source type to ZegoVideoSourceTypeScreenCapture and start screen sharing to push screen sharing video data to the SDK. For details, please refer to Screen Sharing.

Call the setAudioSource interface to set the audio capture source.

Set audio capture source only

/** Use microphone as audio capture source */
[[ZegoExpressEngine sharedEngine] setAudioSource:ZegoAudioSourceTypeMicrophone];

/** Use microphone as audio capture source */
[[ZegoExpressEngine sharedEngine] setAudioSource:ZegoAudioSourceTypeMicrophone];

Set audio capture source and mix configuration at the same time

ZegoAudioSourceMixConfig *audioSourceMixConfig = [ZegoAudioSourceMixConfig defaultConfig];

/** Mix the sound played by the media player with index 0 */
audioSourceMixConfig.mediaPlayerCount = 1;
audioSourceMixConfig.mediaPlayerIndexList = (int *)malloc(sizeof(int)*1);
audioSourceMixConfig.mediaPlayerIndexList[0] = 0;

/** Mix the sound played by the audio effect player with index 0 */
audioSourceMixConfig.audioEffectPlayerCount = 1;
audioSourceMixConfig.audioEffectPlayerIndexList = (int *)malloc(sizeof(int)*1);
audioSourceMixConfig.audioEffectPlayerIndexList[0] = 0;

/** Use microphone as audio capture source and set mix configuration */
[[ZegoExpressEngine sharedEngine] setAudioSource:ZegoAudioSourceTypeMicrophone config:audioSourceMixConfig];

free(audioSourceMixConfig.mediaPlayerIndexList);
free(audioSourceMixConfig.audioEffectPlayerIndexList);

ZegoAudioSourceMixConfig *audioSourceMixConfig = [ZegoAudioSourceMixConfig defaultConfig];

/** Mix the sound played by the media player with index 0 */
audioSourceMixConfig.mediaPlayerCount = 1;
audioSourceMixConfig.mediaPlayerIndexList = (int *)malloc(sizeof(int)*1);
audioSourceMixConfig.mediaPlayerIndexList[0] = 0;

/** Mix the sound played by the audio effect player with index 0 */
audioSourceMixConfig.audioEffectPlayerCount = 1;
audioSourceMixConfig.audioEffectPlayerIndexList = (int *)malloc(sizeof(int)*1);
audioSourceMixConfig.audioEffectPlayerIndexList[0] = 0;

/** Use microphone as audio capture source and set mix configuration */
[[ZegoExpressEngine sharedEngine] setAudioSource:ZegoAudioSourceTypeMicrophone config:audioSourceMixConfig];

free(audioSourceMixConfig.mediaPlayerIndexList);
free(audioSourceMixConfig.audioEffectPlayerIndexList);

Note

If you need to use custom audio capture as the audio capture source, you need to set the audio capture source type to ZegoAudioSourceTypeCustom, and call interfaces such as sendCustomAudioCaptureAACData or sendCustomAudioCapturePCMData to send audio data to the SDK. For details, please refer to Advanced Features - Custom Audio Capture and Rendering.
If you need to use a media player as the audio capture source, you need to set the audio capture source type to ZegoAudioSourceTypeMediaPlayer, call the createMediaPlayer interface to create a media player, and then use the newly created media player to load resources and play. For details, please refer to Media Player.

For the process of logging in to the room and publishing stream, please refer to Login to Room and Publish Stream in implementing video call.

4 (Optional) Switch Audio/Video Capture Sources

Call the setVideoSource interface to switch the video capture source.

/** Use screen sharing as video capture source */
[[ZegoExpressEngine sharedEngine] setVideoSource:ZegoVideoSourceTypeScreenCapture];

/** Use screen sharing as video capture source */
[[ZegoExpressEngine sharedEngine] setVideoSource:ZegoVideoSourceTypeScreenCapture];

Call the setAudioSource interface to switch the audio capture source.

/** Use media player as audio capture source */
[[ZegoExpressEngine sharedEngine] setAudioSource:ZegoAudioSourceTypeMediaPlayer];

/** Use media player as audio capture source */
[[ZegoExpressEngine sharedEngine] setAudioSource:ZegoAudioSourceTypeMediaPlayer];

5 Stop Publishing Stream

For the process of stopping publishing stream, please refer to Stop Streaming in implementing video call.

/** Stop publishing stream */
[[ZegoExpressEngine sharedEngine] stopPublishingStream];

/** Stop publishing stream */
[[ZegoExpressEngine sharedEngine] stopPublishingStream];

FAQ

How to capture screen video and system audio?

If you need to capture screen video and system audio, please refer to Screen Sharing.

Does it support setting multiple media players?

Yes, but note that the same player instance can only be occupied by one channel.

What are the limitations when combining video sources and audio sources?

The limitations for combining video sources and audio sources are as follows:

The main channel cannot use the media player's video or audio source.
When the auxiliary channel uses a media player as the audio/video source, the main channel needs to use and start the microphone device (enableMicrophone). If you don't need microphone sound on the main channel, you can disable audio publishing through mutePublishStreamAudio to keep the microphone started.
Only when the main channel uses camera video source or microphone audio source, the auxiliary channel copying the main channel's video source or audio source can take effect.
The main channel's audio source cannot be set to None, otherwise audio will not be able to render.