What is Stream Mixing?

Stream mixing is a technology that combines multiple audio or video streams into one stream on the cloud. Scenarios like live streaming, online education, and live audio rooms, widely use this technology. Developers can view the screens and hear the voice of all members in a room by playing the mixed stream without managing every single stream in the room.

ZEGOCLOUD, a global RTC products and services provider, supports using this technology with different methods.

Benefits of Stream Mixing

It comes with a lot of benefits for developers.

Low costs

Multiple parties co-host in a room in large-scale live streaming or online education scenarios. Usually, users in the room need to play various streams; however, by mixing streams they can play only one stream.

This means that it reduces the costs by half. In the case of multi-party co-hosting, the costs can be reduced by (n - 1)/n.

Simple code logic

Let’s take the multiple hosts cohosting scenario. Playing and rendering various streams is no longer necessary with Stream Mixing. The audience only needs to play the mixed stream and render it.

Easy forwarding between different platforms

Without it, we cannot forward live streaming to Facebook, YouTube, and other live streaming platforms in multi-party co-hosting scenarios. In fact, these platforms only have one RTMP add and press, and we cannot forward multiple streams to one address.

Support by web browsers on mobile clients

Safari does not support the simultaneous playing of multiple audio files for iPhone users. Therefore, Stream Mixing can solve this problem.

Due to the limit of mobile phone performance and browser performance, most web browsers on mobile phones can play up to four streams generally.

With this technology, the maximum number of playable streams increases dramatically without extra bandwidth or performance consumption.

How does Stream Mixing work?

As shown in the following figure, when multiple users in a room publish streams, the server combines the two streams into one stream based on layout configuration. The audience plays the mixed stream to view the screens of user A and user B.

Stream mixing implementation

Stream mixing process

The server listens for stream changes in a room.
The host on a client publishes a stream.
When the server detects the first new stream, it starts stream mixing.
Co-hosting starts to publish streams.
The server detects stream addition and updates the stream mixing layout configuration.
Co-hosting stops.
The server detects stream reduction and updates the stream mixing layout configuration.
The room is disbanded, and the stream mixing task stops.

Client logic

The client does not need to manage the logic.

It only needs to determine whether to play the original stream or the mixed stream based on whether it needs to publish streams, as shown in the following figure.

Server logic

The server needs to listen for stream changes in the room and updates the Stream Mixing configuration when adding or reducing a stream.

According to requirements, the layout configuration varies with the number of streams. Stream Mixing will stop when the number of streams is 0.

Layout configuration

ZEGOCLOUD provides a layout configuration API. Developers only need to set the position and size of each stream. The following examples show some sample codes.

In the following examples, the resolution of the video screen layout is 360 × 640.

1. Two views side by side

/** Create an input stream list. */
ArrayList<ZegoMixerInput> inputList = new ArrayList<>();

/** Configue the first input stream, including the streamID (the real ID of the input stream), input type, layout, etc.*/
ZegoMixerInput input_1 = new ZegoMixerInput("streamID_1", ZegoMixerInputContentType.VIDEO, new Rect(0, 0, 180, 640));
inputList.add(input_1);
/** Configue the second input stream. */
ZegoMixerInput input_2 = new ZegoMixerInput("streamID_2", ZegoMixerInputContentType.VIDEO, new Rect(180, 0, 360, 640)); 
inputList.add(input_2);

/** Set up the input steam list for the stream mixing task. */
task.setInputList(inputList);

2. Four ideas tiled vertically

/** Create an input stream list.  */
ArrayList<ZegoMixerInput> inputList = new ArrayList<>();

/** Configue the first input stream, including the streamID (the real ID of the input stream), input type, layout, etc.*/
ZegoMixerInput input_1 = new ZegoMixerInput("streamID_1", ZegoMixerInputContentType.VIDEO, new Rect(0, 0, 180, 320));
inputList.add(input_1);
/** Configue the second input stream. */
ZegoMixerInput input_2 = new ZegoMixerInput("streamID_2", ZegoMixerInputContentType.VIDEO, new Rect(180, 0, 360, 320));
inputList.add(input_2);
/** Configue the third input stream. */
ZegoMixerInput input_3 = new ZegoMixerInput("streamID_3", ZegoMixerInputContentType.VIDEO, new Rect(0, 320, 180, 640));
inputList.add(input_3);
/** Configue the fourth input stream. */
ZegoMixerInput input_4 = new ZegoMixerInput("streamID_4", ZegoMixerInputContentType.VIDEO, new Rect(180, 320, 360, 640));
inputList.add(input_4);

/** Set up the input steam list for the stream mixing task. */
task.setInputList(inputList);

One large view tiled with two small views suspended

The position of the input stream in the input stream list determines the layer level of the input stream. The further the order in the list, the higher the layer level.

As shown in the code below, the layer of input stream 2 and input stream 3 is higher than that of input stream 1, so streams 2 and 3 hover over the screen of input stream 1.

/** Create an input stream list.  */
ArrayList<ZegoMixerInput> inputList = new ArrayList<>();

/** Configue the first input stream, including the streamID (the real ID of the input stream), input type, layout, etc.*/
ZegoMixerInput input_1 = new ZegoMixerInput("streamID_1", ZegoMixerInputContentType.VIDEO, new Rect(0, 0, 360, 640));
inputList.add(input_1);
/** Configue the second input stream. */
ZegoMixerInput input_2 = new ZegoMixerInput("streamID_2", ZegoMixerInputContentType.VIDEO, new Rect(230, 200, 340, 400));
inputList.add(input_2);
/** Configue the third input stream. */
ZegoMixerInput input_3 = new ZegoMixerInput("streamID_3", ZegoMixerInputContentType.VIDEO, new Rect(230, 420, 340, 620));
inputList.add(input_3);

/** Set up the input steam list for the stream mixing task. */
task.setInputList(inputList);

For a detailed layout description, see ZEGOCLOUD Document.