Documentation
Live Streaming
Upgrade the livestream
Advanced features
Mix the live streams

Mix the video streams

Last updated：2025-05-07 18:39

Introduction

Through the stream-mixing service, multiple published media streams can compose a single stream , which allows audiences to play just one stream to improve quality and reduce performance cost.

Prerequisites

Before you begin to use stream mixing in your project, make sure you complete the following steps:

Create a project in ZEGOCLOUD Admin Console, and get the AppID and ServerSecret of your project.
Integrate the Video Call SDK into the project. For details, see Integration and Implementation.
Contact us to activate the Stream Mixing service.

Manual stream mixing

With the manual stream mixing, you can customize the stream mixing tasks. which is often used for multiple users to co-host an interactive live streaming event and co-hosting across rooms.

To implement the stream mixing with SDK or ZEGO server API, refer to the Start stream mixing and Stop stream mixing.

The following describes how to implement the manual stream mixing in detail.

Set up a stream mixing task

To set up a stream mixing task, create a ZegoMixStreamConfig object and set up the configuration parameters accordingly.

1. Set up the task ID

The TaskID property of the ZegoMixStreamConfig object is a custom-defined ID of the stream mixing task, which must be unique.
The SDK uses the order of the input streams in the input stream list to determine the layout of the mixed stream. The streams are stacked on top of each other. To avoid larger layouts covering smaller layouts, the input streams of larger layouts should be placed before the input streams of smaller layouts in the list. For example, if you want to place a smaller layout B on top of layout A, the input stream list should be written as ZegoMixStreamInput[] inputStreamInfo = {streamInfoA, streamInfoB}.
When all the "ContentType" of the input streams for mixing are set to "AUDIO", the SDK does not process the layout field internally, so you don't need to pay attention to the "layout" parameter.
When all the "ContentType" of the input streams for mixing are set to "AUDIO", the SDK will default the resolution to 1*1 (i.e., pure audio output for the mixed stream). If you want the mixed stream to have video frames or a background image, you need to set the "ContentType" of at least one input stream to "VIDEO".

2. Set up the input streams

According to the actual business scenario, call the ZegoMixStreamConfig with the definition of inputList to input the list of video streams. The list contains objects of ZegoMixStreamInput, representing streams with "streamID". Build N ZegoMixStreamInput objects and put them into the input stream list.

"taskID" is the ID of the mixing stream task, which is defined by the developer and must be unique.
The SDK uses the order of input streams in the input stream list to determine the layout of the mixed stream. The streams are stacked in the order they appear in the list. To avoid larger layouts covering smaller layouts, the input streams of larger layouts should be placed before the input streams of smaller layouts in the list. For example, if you want to place a smaller layout B on top of a layout A, the input stream list should be written as ZegoMixStreamInput[] inputStreamInfo = {streamInfoA, streamInfoB}.
When the "ContentType" of all input streams for mixing is set to "AUDIO", the SDK does not process the layout field internally, so you don't need to pay attention to the "layout" parameter.
When the "ContentType" of all input streams for mixing is set to "AUDIO", the SDK will default the resolution to 1*1 (i.e., pure audio output for mixing). If you want the mixed stream output to have video frames or a background image, you need to set the "ContentType" of at least one input stream to "VIDEO".

Example of using ZegoMixStreamInput:

The layout of the input stream is based on the coordinate system with the upper-left corner of the output mixed stream as the origin. Set the layout of the input stream based on the reference origin, which is passed as the "layout" parameter. In addition, the layer order of the input stream is determined by its position in the input stream list. The later the position, the higher the layer order.

The explanation of the "layout" parameter is as follows:

Parameter	Description
left	The x-coordinate of the upper-left corner of the input stream frame.
top	The y-coordinate of the upper-left corner of the input stream frame.
right	The x-coordinate of the lower-right corner of the input stream frame.
bottom	The y-coordinate of the lower-right corner of the input stream frame.

Assuming that the upper-left corner of a certain stream is at coordinates (50, 300) and the lower-right corner is at coordinates (200, 450), the position of this stream in the final output mixed stream is shown in the following diagram:

 /**
     * The origin is at the upper left corner, top/bottom/left/right is defined as follows:
     *
     * (left, top)-----------------------
     * | |
     * | |
     * | |
     * | |
     * -------------------(right, bottom)
     */

// inputList parameter settings
const inputList = [
    // Screen A
    {
        streamID: "streamA",
        layout: {
            top: 0,
            left: 0,
            bottom: 375,
            right: 667,
        },
    },
    // Screen B
    {
        streamID: "streamB",
        layout: {
            top: 300,
            left: 50,
            bottom: 450,
            right: 200,
        },
    },

];

Developers can refer to the following sample code to implement common mixing layouts: two screens tiled horizontally, four screens tiled horizontally and vertically, one large screen filling the layout, and two small screens floating.

The following layout examples are based on a resolution of 360×640.

Mixing layout example 1: two screens tiled horizontally

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
    },
    {
        streamID: "streamB", // Right stream
        layout: {
            top: 0,
            left: 180,
            bottom: 640,
            right: 360,
        },
    }
]
// Start mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Mixing layout example 2: four screens tiled horizontally and vertically

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Top-left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 320,
            right: 180,
        },
    },
    {
        streamID: "streamC", // Bottom-left stream
        layout: {
            top: 320,
            left: 0,
            bottom: 640,
            right: 180,
        },
    },
    {
        streamID: "streamB", // Top-right stream
        layout: {
            top: 0,
            left: 180,
            bottom: 320,
            right: 360,
        },
    },
    {
        streamID: "streamD", // Bottom-right stream
        layout: {
            top: 320,
            left: 180,
            bottom: 640,
            right: 360,
        },
    }
]
// Start mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Mixing layout example 3: one large screen filling the layout and two small screens floating

The layer order of the input streams is determined by their position in the inputList. The higher the position in the list, the higher the layer order. In the example code below, the layer order of the second input stream and the third input stream is higher than the first input stream, so the second and third streams float above the first stream's screen.

// Set inputList
const inputList = [
    {
        streamID: "streamA", // First stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 320,
        },
    },
    {
        streamID: "streamB", // Second stream
        layout: {
            top:200,
            left:230,
            bottom:400,
            right: 340,
        },
    },
    {
        streamID: "streamC", // Third stream
        layout: {
            top:420,
            left:230,
            bottom:620,
            right: 340,
        },
    }
]
// Start mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Set up the mixing outputs

The target property of the outputList parameter can be a stream ID or a URL. If the mixed stream needs to be published to CDN, you must use a URL as the output target. Otherwise, it is up to you to use a URL or a stream ID. The "outputConfig" parameter can be used to set the resolution, frame rate, bit rate, audio encoding, and other settings for the output stream.

The maximum frame rate for mixed stream output is limited to 20 frames by default. If you need to output a higher frame rate, please contact technical support for configuration.
If you need to play mixed stream CDN resources on the web, when using CDN recording, please select AAC-LC as the audio codec. Some browsers (such as Google Chrome and Microsoft Edge) are not compatible with the HE-AAC audio codec format, which may cause the recorded files to be unplayable.

const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set the mixed stream output
    outputList:['mixStreamID'],

    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

(Optional) Set image watermark for mixing

Setting Image Watermark for Mixing

If you need the URL of the watermark image, please contact ZEGOCLOUD technical support to obtain it.

The following code demonstrates setting a ZEGOCLOUD image watermark placed in the top left corner of the screen:

// Set the watermark image
const watermark = [
    {
        imageURL: "preset-id://zegowp.png",
        layout: {
            top: 0,
            left: 0,
            bottom: 100,
            right: 200,
        },
    }
]
// Start mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    watermark,
    // Set the mixed stream output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

(Optional) Set mixed stream background image

Set mixed stream background image

If you need the URL of the background image ZegoMixerImageInfo, please contact ZEGOCLOUD technical support to obtain it, or you can use a network image, but only HTTP protocol is supported.

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
        // Set mixed stream background image
        imageInfo: {
            url: "preset-id://zegobg.png",
            displayMode: 1, // Display the background image only when the camera is turned off. You can check the interface documentation for specific parameter values.
        }
    }
]
// Start mixing stream
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set mixed stream output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

(Optional) Set mixed stream sound level callback

Set mixed stream sound level callback

In video scenarios, it is not recommended to turn on the sound wave switch, otherwise the web pulling HLS protocol stream may have compatibility issues.

You can choose whether to enable the sound level callback of the mixed stream by setting the enableSoundLevel parameter. After enabling it (set to "True"), when the user pulls the mixed stream, they can receive the volume change (sound level) information of each individual stream through the mixerSoundLevelUpdate callback.

// Start mixing stream
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    // Enable the sound level callback of the mixed stream. After enabling it, you can receive the sound level information of each individual stream through the [onMixerSoundLevelUpdate] callback when pulling the mixed stream.
    enableSoundLevel: true,
    inputList: {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        }, 
        soundLevelID: 1, // Mixed stream sound level ID, used to find the corresponding sound level value of the input stream in [mixerSoundLevelUpdate].
    },
    // Set mixed stream output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Start the stream mixing task

After the stream publishing starts successfully, call startMixerTask to send the mixing configuration to the ZEGOCLOUD mixing server and trigger the mixing process.

const result = await zg.startMixerTask(config)
if(result.errorCode !== 0) {
    // Mixing failed
    console.error(result.extendedData)
}

If you want to mix pure audio data, some parameter configurations will have the following special handling:

For the input stream "inputList", the pure audio configuration rules are: "top", "left", "bottom", "right" cannot all be set to "0" or all be set to "1". Recommended configuration: "top = 0, left = 0, bottom = 1, right = 1".
For the "outputConfig" pure audio configuration, the rules are: "outputFPS" and "outputBitrate" cannot be set to "0". Recommended configuration: "outputFPS = 1, outputBitrate = 1, outputResolution = (1,1)".

Stream mixing callback

When there are changes in the mixed stream information, such as adding or removing input stream lists for mixing, adjusting the output bitrate of the mixed video, you can modify the parameters of the mixer task object and then call startMixerTask again to update the mixing configuration on the ZEGOCLOUD mixing server.

When updating the configuration of a transcoding task, the "taskID" cannot be changed.

The following are the common errors of a stream mixing task:

Error code	Description
errorCode = 150	Input stream does not exist.
errorCode = 151	Stream mixing failed.
errorCode = 152	Failed to stop stream mixing.
errorCode = 153	Incorrect input parameters.
errorCode = 154	Incorrect output parameters.
errorCode = 155	Incorrect input resolution format.
errorCode = 156	Incorrect output resolution format.
errorCode = 157	Steam mixing not enabled.

Stop the stream mixing task

To stop a stream mixing task, call the stopMixerTask method.

The task ID used for starting and stopping the stream mixing must be the same.

try {
    await zg.stopMixerTask(taskID);
    alert('Stop mixing stream successfully...');
} catch (err) {
    alert('Failed to stop mixing stream...');
    console.log('stopMixStream err: ', err);
}