Stream Mixing

Function Overview

Stream Mixing is a technology that merges multiple audio and video streams into a single stream from the cloud, also known as stream mixing. Developers only need to play the merged stream to see all room members' video and hear all room members' audio, without needing to manage each stream in the room separately.

This document mainly describes how to initiate stream mixing from the client. If you need to initiate stream mixing from your own server, please refer to Server API - Start Mix.

Stream Mixing Method Classification

ZEGOCLOUD supports Manual Stream Mixing, Automatic Stream Mixing, and Fully Automatic Stream Mixing. The differences between the three methods are as follows:

Advantages of Stream Mixing

1. Reduces development complexity. For example, when N hosts are co-hosting, if stream mixing is used, the audience doesn't need to play N video streams simultaneously, eliminating the need to develop logic for playing N streams and layout.
2. Reduces device performance requirements, minimizing device performance overhead and network bandwidth burden. For example, when there are too many co-hosts, the audience needs to play N video streams, requiring device hardware to support playing N streams simultaneously.
3. Relaying to multiple CDNs is simple, just add output streams as needed during stream mixing configuration.
4. When the audience needs to replay multi-host co-hosting videos, only need to enable recording configuration on the CDN.
5. Content moderation only needs to observe one screen, no need to view multiple screens simultaneously.

Download Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please view files in the "src/Examples/Others/StreamMixing" directory.

Prerequisites

Before implementing stream mixing functionality, please ensure:

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and Server address. For details, please refer to Console - Project Information.
• You have integrated the ZEGO Express SDK into the project and implemented basic audio/video stream publishing and playing functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Manual Stream Mixing Usage Steps

Manual stream mixing allows custom control of stream mixing tasks and content, including input streams, mixing layout, etc. It is commonly used in multi-user interactive live streaming and cross-room co-hosting scenarios. Supports manual mixing of video and audio streams.

Developers can implement manual stream mixing functionality through SDK or ZEGOCLOUD server API. For server-related interfaces, please refer to Start Mix and Stop Mix.

The following describes how to implement manual stream mixing using the SDK.

The main process of stream mixing:

1. Join a room and publish/play streams.
2. Start stream mixing.
3. Distribute the URL address after successful stream mixing through the backend.
4. The audience plays the stream after obtaining the URL address.

According to the above process, the SDK API call process is as follows.

Publish Stream

Publishing stream is a prerequisite for stream mixing. For details, please refer to "Publish stream" in Quick Start - Implementation.

Set Stream Mixing Configuration

Set Stream Mixing Input Streams

According to the actual business scenario, call inputList in ZegoMixStreamConfig to define the list of input video streams. The list contains ZegoMixStreamInput objects, representing streams with "streamID". Construct N ZegoMixStreamInput objects according to the required number of input streams and place them in the input stream list.

Usage example of ZegoMixStreamInput:

The layout of an input stream uses the upper-left corner of the output mixed stream screen as the origin of the coordinate system. Set the input stream layout relative to the origin, i.e., pass in the "layout" parameter of the input stream. Additionally, the layer hierarchy of input streams is determined by their position in the input stream list - the later the position in the list, the higher the layer hierarchy.

The "layout" parameter description is as follows:

Assuming the upper-left corner coordinate of a stream is specified as (50, 300) and the lower-right corner coordinate is (200, 450), the position of this stream in the final output mixed stream is shown in the following figure:

 /**
     *  Origin in the upper-left corner, top/bottom/left/right defined as follows:
     *
     *  (left, top)-----------------------
     *  |                                |
     *  |                                |
     *  |                                |
     *  |                                |
     *  -------------------(right, bottom)
     */

// inputList parameter settings
const inputList = [
    // Screen A
    {
        streamID: "streamA",
        layout: {
            top: 0,
            left: 0,
            bottom: 375,
            right: 667,
        },
    },
    // Screen B
    {
        streamID: "streamB",
        layout: {
            top: 300,
            left: 50,
            bottom: 450,
            right: 200,
        },
    },

];

 /**
     *  Origin in the upper-left corner, top/bottom/left/right defined as follows:
     *
     *  (left, top)-----------------------
     *  |                                |
     *  |                                |
     *  |                                |
     *  |                                |
     *  -------------------(right, bottom)
     */

// inputList parameter settings
const inputList = [
    // Screen A
    {
        streamID: "streamA",
        layout: {
            top: 0,
            left: 0,
            bottom: 375,
            right: 667,
        },
    },
    // Screen B
    {
        streamID: "streamB",
        layout: {
            top: 300,
            left: 50,
            bottom: 450,
            right: 200,
        },
    },

];

Developers can refer to the following example code to implement common stream mixing layouts: two screens horizontally tiled, four screens horizontally and vertically tiled, one large screen filled and two small screens floating.

The following layout examples are all explained using 360×640 resolution.

Stream Mixing Layout Example 1: Two Screens Horizontally Tiled

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
    },
    {
        streamID: "streamB", // Right stream
        layout: {
            top: 0,
            left: 180,
            bottom: 640,
            right: 360,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
    },
    {
        streamID: "streamB", // Right stream
        layout: {
            top: 0,
            left: 180,
            bottom: 640,
            right: 360,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Stream Mixing Layout Example 2: Four Screens Horizontally and Vertically Tiled

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Upper-left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 320,
            right: 180,
        },
    },
    {
        streamID: "streamC", // Lower-left stream
        layout: {
            top: 320,
            left: 0,
            bottom: 640,
            right: 180,
        },
    },
    {
        streamID: "streamB", // Upper-right stream
        layout: {
            top: 0,
            left: 180,
            bottom: 320,
            right: 360,
        },
    },
    {
        streamID: "streamD", // Lower-right stream
        layout: {
            top: 320,
            left: 180,
            bottom: 640,
            right: 360,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Upper-left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 320,
            right: 180,
        },
    },
    {
        streamID: "streamC", // Lower-left stream
        layout: {
            top: 320,
            left: 0,
            bottom: 640,
            right: 180,
        },
    },
    {
        streamID: "streamB", // Upper-right stream
        layout: {
            top: 0,
            left: 180,
            bottom: 320,
            right: 360,
        },
    },
    {
        streamID: "streamD", // Lower-right stream
        layout: {
            top: 320,
            left: 180,
            bottom: 640,
            right: 360,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Stream Mixing Layout Example 3: One Large Screen Filled and Two Small Screens Floating

The layer hierarchy of input streams is determined by their position in the input stream list. The later the position in the list, the higher the layer hierarchy. As shown in the following example code, the layer hierarchy of the 2nd input stream and the 3rd input stream is higher than that of the 1st input stream, so the 2nd and 3rd streams float on top of the 1st stream's screen.

// Set inputList
const inputList = [
    {
        streamID: "streamA", // First stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 320,
        },
    },
    {
        streamID: "streamB", // Second stream
        layout: {
            top:200,
            left:230,
            bottom:400,
            right: 340,
        },
    },
    {
        streamID: "streamC", // Third stream
        layout: {
            top:420,
            left:230,
            bottom:620,
            right: 340,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

// Set inputList
const inputList = [
    {
        streamID: "streamA", // First stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 320,
        },
    },
    {
        streamID: "streamB", // Second stream
        layout: {
            top:200,
            left:230,
            bottom:400,
            right: 340,
        },
    },
    {
        streamID: "streamC", // Third stream
        layout: {
            top:420,
            left:230,
            bottom:620,
            right: 340,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

Set Stream Mixing Output

In the stream mixing parameter "outputList", the output stream's target can be a stream name "streamID" or "URL". If you need to publish the stream to CDN, use "URL"; if you don't need to publish the stream to CDN, the caller can decide whether to use "URL" or "streamID". The "outputConfig" parameter can set the output stream's resolution, frame rate, bitrate, audio encoding, etc.

const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],

    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],

    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

(Optional) Set Stream Mixing Image Watermark

// Set watermark image
const watermark = [
    {
        imageURL: "preset-id://zegowp.png",
        layout: {
            top: 0,
            left: 0,
            bottom: 100,
            right: 200,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    watermark,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

// Set watermark image
const watermark = [
    {
        imageURL: "preset-id://zegowp.png",
        layout: {
            top: 0,
            left: 0,
            bottom: 100,
            right: 200,
        },
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    watermark,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
})

(Optional) Set Stream Mixing Background Image

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
        // Set stream mixing background image
        imageInfo: {
            url: "preset-id://zegobg.png",
            displayMode: 1, // Display background image only when camera is off. For specific parameter values, please refer to the interface documentation
        }
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
}

// Set inputList
const inputList = [
    {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
        // Set stream mixing background image
        imageInfo: {
            url: "preset-id://zegobg.png",
            displayMode: 1, // Display background image only when camera is off. For specific parameter values, please refer to the interface documentation
        }
    }
]
// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    inputList,
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
}

(Optional) Set Stream Mixing Sound Level Callback

// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    // Enable stream mixing sound level callback notification. After enabling, you can receive sound level information of each single stream through [onMixerSoundLevelUpdate] callback when playing the mixed stream
    enableSoundLevel: true,
    inputList: {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
        soundLevelID: 1, // Stream mixing sound level ID, used to find the sound level value of the corresponding input stream in [mixerSoundLevelUpdate].
    },
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
}

// Start stream mixing
const result = await zg.startMixerTask({
    taskID:'taskUUID',
    // Enable stream mixing sound level callback notification. After enabling, you can receive sound level information of each single stream through [onMixerSoundLevelUpdate] callback when playing the mixed stream
    enableSoundLevel: true,
    inputList: {
        streamID: "streamA", // Left stream
        layout: {
            top: 0,
            left: 0,
            bottom: 640,
            right: 180,
        },
        soundLevelID: 1, // Stream mixing sound level ID, used to find the sound level value of the corresponding input stream in [mixerSoundLevelUpdate].
    },
    // Set stream mixing output
    outputList:['mixStreamID'],
    outputConfig: {
       outputBitrate: 300,
       outputFPS: 15,
       outputWidth: 360,
       outputHeight: 640,
    }
}

Start Stream Mixing

After successfully publishing the stream, call startMixerTask to send the stream mixing configuration to the ZEGOCLOUD stream mixing server and trigger stream mixing.

const result = await zg.startMixerTask(config)
if(result.errorCode !== 0) {
    // Stream mixing failed
    console.error(result.extendedData)
}

const result = await zg.startMixerTask(config)
if(result.errorCode !== 0) {
    // Stream mixing failed
    console.error(result.extendedData)
}

Stream Mixing Configuration Update Notification

When stream mixing information changes, such as adding or removing input streams in the stream mixing input list, adjusting stream mixing video output bitrate, etc., modify the parameters of that stream mixing task object, then call startMixerTask once to update the stream mixing configuration on the ZEGOCLOUD stream mixing server.

Common error codes and their meanings are as follows:

Stop Stream Mixing

Stopping stream mixing is achieved by calling stopMixerTask.

try {
    await zg.stopMixerTask(taskID);
    alert('Stop stream mixing succeeded...');
} catch (err) {
    alert('Stop stream mixing failed...');
    console.log('stopMixStream err: ', err);
}

try {
    await zg.stopMixerTask(taskID);
    alert('Stop stream mixing succeeded...');
} catch (err) {
    alert('Stop stream mixing failed...');
    console.log('stopMixStream err: ', err);
}

Automatic Stream Mixing Usage Steps

Initialize and Login to Room

Please refer to "Create engine" and "Login room" in Quick Start - Implementation.

Set Stream Mixing Configuration, Start Automatic Stream Mixing

ZegoAutoMixerTask is an automatic stream mixing task configuration object defined in the SDK. By configuring this object, you can customize automatic stream mixing tasks.

Set Stream Mixing Configuration, Start Automatic Stream Mixing

Create a new automatic stream mixing task object, then set input, output, and other parameters separately.

• There can only be one automatic stream mixing task ID in a room, ensuring the uniqueness of the automatic stream mixing task ID. It is recommended to associate the automatic stream mixing task ID with the room ID, and you can directly use the room ID as the automatic stream mixing task ID.
• The room ID that needs automatic stream mixing. If the room does not exist, automatic stream mixing cannot be performed.

zg.startAutoMixerTask({
    taskID, // Automatic stream mixing task ID
    roomID: "room1", // Room ID for automatic stream mixing task
    audioConfig: {
        bitrate: 64,
        channel: 2,
        codecID: 6
    },
    outputList: [
        // Automatic stream mixing output list
        {
            target: mixStreamID
        }
    ],
    enableSoundLevel: true
})
    .then(res => {
        console.warn("startAutoMixer", res);
    })
    .catch(e => {
        console.error("startAutoMixer", e);
    });
}

zg.startAutoMixerTask({
    taskID, // Automatic stream mixing task ID
    roomID: "room1", // Room ID for automatic stream mixing task
    audioConfig: {
        bitrate: 64,
        channel: 2,
        codecID: 6
    },
    outputList: [
        // Automatic stream mixing output list
        {
            target: mixStreamID
        }
    ],
    enableSoundLevel: true
})
    .then(res => {
        console.warn("startAutoMixer", res);
    })
    .catch(e => {
        console.error("startAutoMixer", e);
    });
}

(Optional) Set Automatic Stream Mixing Audio Configuration

zg.startAutoMixerTask({
    taskID, // Automatic stream mixing task ID
    roomID: "room1", // Room ID for automatic stream mixing task
    // Audio configuration
    audioConfig: {
        bitrate: 64, // Audio bitrate
        channel: 2, // Number of audio channels
        codecID: 6
    },
    outputList: [
        // Automatic stream mixing output list
        {
            target: mixStreamID
        }
    ]
})
    .then(res => {
        console.warn("startAutoMixer", res);
    })
    .catch(e => {
        console.error("startAutoMixer", e);
    });

zg.startAutoMixerTask({
    taskID, // Automatic stream mixing task ID
    roomID: "room1", // Room ID for automatic stream mixing task
    // Audio configuration
    audioConfig: {
        bitrate: 64, // Audio bitrate
        channel: 2, // Number of audio channels
        codecID: 6
    },
    outputList: [
        // Automatic stream mixing output list
        {
            target: mixStreamID
        }
    ]
})
    .then(res => {
        console.warn("startAutoMixer", res);
    })
    .catch(e => {
        console.error("startAutoMixer", e);
    });

(Optional) Set Automatic Stream Mixing Sound Level Callback

zg.startAutoMixerTask({
    taskID, // Automatic stream mixing task ID
    roomID: "room1", // Room ID for automatic stream mixing task
    outputList: [
        {
            target: mixStreamID
        }
    ],
    enableSoundLevel: true
})
    .then(res => {
        console.warn("startAutoMixer", res);
    })
    .catch(e => {
        console.error("startAutoMixer", e);
    });

zg.startAutoMixerTask({
    taskID, // Automatic stream mixing task ID
    roomID: "room1", // Room ID for automatic stream mixing task
    outputList: [
        {
            target: mixStreamID
        }
    ],
    enableSoundLevel: true
})
    .then(res => {
        console.warn("startAutoMixer", res);
    })
    .catch(e => {
        console.error("startAutoMixer", e);
    });

Stop Automatic Stream Mixing

Call the stopAutoMixerTask interface to stop automatic stream mixing.

// Pass in the previously created stream mixing task object
zg.stopAutoMixerTask({
    taskID: taskID,
    roomID: "room1",
})
    .then(res => {
        console.warn("stopAutoMixerTask", res);
    })
    .catch(e => {
        console.error("stopAutoMixerTask", e);
    });

// Pass in the previously created stream mixing task object
zg.stopAutoMixerTask({
    taskID: taskID,
    roomID: "room1",
})
    .then(res => {
        console.warn("stopAutoMixerTask", res);
    })
    .catch(e => {
        console.error("stopAutoMixerTask", e);
    });

Fully Automatic Stream Mixing Usage Steps

Implement automatic audio stream mixing for each room through ZEGOCLOUD server-side configuration. For details, please contact ZEGOCLOUD Technical Support.