Stream mixing

Introduction

Through the stream-mixing service, multiple published media streams can compose a single stream , which allows audiences to play just one stream to improve quality and reduce performance cost.

ZEGOCLOUD supports using stream mixing with three methods, manual stream mix, automatic stream mix, and full-automatic stream mix. The following table shows their differences:

Use cases

You can use stream mixing for various use cases. The following are some typical examples:

• When client devices are not capable of processing the required number of streams simultaneously.
• When it is required to synthesize multiple live videos into one single view. For example, to display the teacher's video and the students' videos in a synthesized view.
• When it is required to synthesize multiple audios into one single stream, especially for the voice chatroom scenarios.

Why stream mixing?

• It reduces the complexity of development. For example, when multiple users co-host an interactive live streaming event, it's much easier to receive and play just one single mixed stream rather than to receive and play multiple streams and do the screen layout for each stream individually.
• It reduces the requirements on the client device's performance and network bandwidth. Having more streams to process means more system resources and network bandwidth are required. With stream mixing, the burden of processing multiple streams is removed from the client side.
• It makes it much easier to forward multiple streams to multiple CDNs. You can achieve this by doing simple output stream configurations for stream mixing tasks.
• When it is required to record multiple streams for future playback, the recording can be done on the CDN to which the mixed stream is forwarded.
• Stream mixing also makes content moderation easier because it can be done by checking the mixed stream rather than checking multiple streams individually.

Prerequisites

Before you begin, make sure you complete the following:

• Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.
• Refer to the Quick Start doc to complete the SDK integration and basic function implementation.
• Contact us to activate the Stream Mixing service.
  

Manual stream mixing

With the manual stream mixing, you can customize the stream mixing tasks. which is often used for multiple users to co-host an interactive live streaming event and co-hosting across rooms.

To implement the stream mixing with SDK or ZEGOCLOUD server API, refer to the Start stream mixing and Stop stream mixing.

The following describes how to implement the manual stream mixing in detail.

Set up a stream mixing task

ZegoMixerTask is a class defined in the SDK for configuring a stream mixing task. The configuration parameters include the input streams, the outputs, and others.

/// Stream mixing task object.
@interface ZegoMixerTask: NSObject

-(instancetype)init NS_UNAVAILABLE;

/// Construct a stream mixing task object using TaskID.
-(instancetype)initWithTaskID:(NSString *)taskID;

/// Task ID of the stream mixing task.
@property (nonatomic, copy, readonly) NSString *taskID;

/// Set up the audio configuration of the mixed stream.
-(void)setAudioConfig:(ZegoMixerAudioConfig *)audioConfig;

/// Set up the video configuration of the mixed stream.
-(void)setVideoConfig:(ZegoMixerVideoConfig *)videoConfig;

/// Set up the list of input streams.
-(void)setInputList:(NSArray<ZegoMixerInput *> *)inputList;

/// Set up the list of mixing outputs.
-(void)setOutputList:(NSArray<ZegoMixerOutput *> *)outputList;

/// Set up the watermark of the mixed stream.
-(void)setWatermark:(ZegoWatermark *)watermark;

/// Set up the background image of the mixed stream.
-(void)setBackgroundImageURL:(NSString *)backgroundImageURL;

@end

1. Create a stream mixing task object

Create a stream mixing task object using the constructor initWithTaskID, and then call the instance methods to set up the input and output parameters.

ZegoMixerTask *task = [[ZegoMixerTask alloc] initWithTaskID:@"task-1"];

// Save the stream mixing task object.
self.mixerTask = task;

2. Optional: Set up the audio configuration of the mixed stream

[task setAudioConfig:[ZegoMixerAudioConfig defaultConfig]];

3. Set up the input streams

For each input stream, create a ZegoMixerInput object.

4. Set up the mixing outputs

A stream mixing task can have up to 3 mixing outputs.

In the following code example, the stream mixing task generates an output with the stream name output-stream, which will be published to the ZEGO server.

NSArray<ZegoMixerOutput *> *outputArray = @[[[ZegoMixerOutput alloc] initWithTarget:@"output-stream"]];
[task setOutputList:outputArray];

5. Optional: Set up the event callback related sound level of the mixed stream

Start the stream mixing task

After configuring the ZegoMixerTask object, call the startMixerTask method to start the stream mixing task. If the stream mixing task doesn't start successfully, handle the failure in the callback that returns the execution result.

• Function prototype:

/// Start the stream mixing task.
/// @param task: The stream mixing task object.
/// @param callback: The callback to return the result of starting up the task.
-(void)startMixerTask:(ZegoMixerTask *)task callback:(nullable ZegoMixerStartCallback)callback;

• Sample code:

[self.engine startMixerTask:task callback:^(ZegoMixerStartResult * _Nonnull result) {
    if (result.errorCode == 0) {
        NSLog(@"Start mixing task success.");
    } else {
        NSLog(@"Start mixing task failed.");
    }
}];

Update the configurations of the stream mixing task

When you need to change the stream mixing task, for example, when the number of input streams increases or decrease, or the bitrate of the output stream needs to be adjusted, you can update the stream mixing task object with the new configurations and then call the startMixerTask method again.

Stop the stream mixing task

To stop a stream mixing task, call the stopMixerTask method, with the corresponding task ID passed to the taskID parameter.

• Function prototype:

/// Stop the stream mixing task.
/// @param taskID: The taskID of the steam mixing task to be stopped.
-(void)stopMixerTask:(NSString *)taskID;

• Sample code:

// Pass in the taskID of the steam mixing task to be stopped.
[self.engine stopMixerTask:self.mixerTask.taskID];

Automatic stream mixing

Set up a stream mixing task

ZegoAutoMixerTask is a class defined in the SDK for configuring a stream mixing task. The configuration parameters include the layout of input streams, the outputs, and others.

/// Stream mixing task object.
@interface ZegoAutoMixerTask : NSObject

/// Task ID of the stream mixing task.
@property (nonatomic, copy) NSString *taskID;

/// Room ID of the stream mixing task.
@property (nonatomic, copy) NSString *roomID;

/// Set up the audio configuration of the mixed stream.
@property (nonatomic, strong) ZegoMixerAudioConfig *audioConfig;

/// Set up the list of mixing outputs.
@property (nonatomic, strong) NSArray<ZegoMixerOutput *> *outputList;

/// Set up the event callback related sound level of the mixed stream.
@property (nonatomic, assign) BOOL enableSoundLevel;

@end

1. Create an auto stream mixing task object

Create an auto stream mixing task object and then call the instance methods to set up the input and output parameters.

• The task ID must be unique within a room. We recommend you set the room ID as the task ID.
• The room ID must have already existed; otherwise, the stream mixing will fail.

ZegoAutoMixerTask *task = [[ZegoAutoMixerTask alloc] init];

task.taskID = @"taskID1";
task.roomID = @"roomID1";

2. Optional: Set up the audio configuration of the auto mixed stream

ZegoMixerAudioConfig *audioConfig = [ZegoMixerAudioConfig init];
audioConfig.bitrate = 48;/** Audio bitrate, unit: kbps, use 48 kbps by default. This can't be changed after stream mixing starts. */
audioConfig.channel = ZegoAudioChannelMono;/** Audio channel, use ZegoAudioChannelMono by default. */
audioConfig.codecID = ZegoAudioCodecIDNormal;/** Codec ID, use ZEGO_AUDIO_CODEC_ID_DEFAULT by default.*/
audioConfig.mixMode = ZegoAudioMixModeRaw;/** Audio mix mode, use ZegoAudioMixModeRaw by default.  */
[task setAudioConfig:audioConfig];

3. Set up the auto mixing outputs

To set up the auto mixing outputs, call the ZegoMixerOutput method to set the output list. And you can play the mixed stream according to the output-stream of the outputs.

NSArray<ZegoMixerOutput *> *outputArray = @[[[ZegoMixerOutput alloc] initWithTarget:@"output-stream"]];/** Mixing output target, URL or streamID. */
[task setOutputList:outputArray];
```tOutputList:outputArray];

4. Optional: Set up the event callback related sound level of the auto mixed stream

Start the auto stream mixing task

After configuring the ZegoAutoMixerTask object, call the startAutoMixerTask method to start the stream mixing task. And the SDK sends out the result notifications through the onAutoMixerSoundLevelUpdate callback.

[[ZegoExpressEngine sharedEngine] startAutoMixerTask:task callback:^(int errorCode, NSDictionary * _Nullable extendedData) {
    if (errorCode == 0) {
        /// Auto stream mixing task started successfully. 
    }
}];

Stop the auto stream mixing task

To stop a stream mixing task, call the stopAutoMixerTask method.

//  Pass in the task object of the steam mixing task to be stopped.
[[ZegoExpressEngine sharedEngine] stopAutoMixerTask:self.autoMixerTask callback:^(int errorCode) {
    if(errorCode == 0) {
        ///Auto stream mixing task stopped successfully.
    }
}];

Full-automatic stream mixing

To fully-automatical mix the audio streams in each room, contact the ZEGOCLOUD Technical Support.

FAQ

1. Can I forward the mixed stream to a third-party CDN or even multiple CDNs?

   Yes. To forward the mixed stream to a third-party CDN, specify the CDN URL in the target property of the ZegoMixerOutput object. The URL must be in RTMP format: rtmp://xxxxxxxx.

   To forward the mixed stream to multiple CDNs, create multiple ZegoMixerOutput objects accordingly, and put them into the outputList of the ZegoMixerTask object.