Video Small-Large Stream and Layered Encoding

2024-04-01

Function Overview

When the following requirements appear in the developer's co-hosting or stream mixing business scenarios, it is recommended to use the Video Layered Encoding or Video Small-Large Stream Encoding functions provided by the SDK:

Need to display video streams of different quality on different terminals.
Need to maintain smooth co-hosting in poor network environments.
Need to adaptively pull the quality of video streams based on network status.

Video Layered Encoding (H.264 SVC) is an extension of H.264 AVC, supporting bitstreams with layered characteristics. Video Small-Large Stream Encoding (H.264 DualStream) refers to the concept of H.264 SVC. By layering video bitstreams, it ensures a smooth experience for terminals with different network and device performance.

Both encoding methods divide the bitstream into "Base Layer" and "Extension Layer", which can provide better experience for users with different network statuses and device performance. The base layer guarantees the most basic video quality, while the extension layer is a supplement to the base layer. For users with good networks, they can pull the extension layer to get a better experience. For users with poor network status, pulling only the base layer can guarantee basic video quality. The following is a detailed comparison of the two functions:

-	Video Layered Encoding (H.264 SVC)	Video Small-Large Stream Encoding (H.264 DualStream)
Implementation Logic	Start 1 encoder to encode bitstreams with different parameters as the base layer and extension layer of the bitstream.	Start 2 encoders to encode bitstreams with different parameters as the base layer and extension layer of the bitstream.
Protocol Used	Use ZEGO's private protocol. The playing end can only pull video streams of different layers from the ZEGO server.
Encoding/Decoding	The encoding processes of the base layer and extension layer are not independent, and can be decoded independently.	The encoding and decoding processes of the base layer and extension layer are both independent of each other.
Publishing Stream	Select the encoding format as "ZegoVideoCodecIDSvc".	Select the encoding format as "ZegoVideoCodecIDH264DualStream". (Optional) Configure the video parameters of the base layer and extension layer separately through the [setPublishDualStreamConfig] interface.
Playing Stream	There is no difference in interface calls. The same user can only pull one layer of the video stream at the same time. By default, when the network is good, only pull the extension layer; when the network is poor, only pull the base layer.
Playing Stream	The resolution of the pulled base layer is 50% of the extension layer, the bitrate is 25% of the extension layer, and the frame rate remains the same. Custom configuration is not possible.	Depends on the video parameters of the base layer and extension layer set by the publishing end.
Advantages	Can generate different bitstreams or extract different bitstreams as needed. Using layered video encoding is more efficient than encoding multiple times with ordinary encoding methods. More flexible application. Stronger network adaptability.	The video parameters of the extension layer and base layer can be configured separately. Can use hardware encoding to reduce CPU performance burden. Better scalability, can support multiple encoding standards in the future. Better universality, mainstream software and hardware encoders all support.
Disadvantages	Slightly lower compression efficiency: Under the same conditions, the compression efficiency of layered video encoding is about 20% lower than ordinary encoding methods. That is, to achieve the same video quality as ordinary encoding methods, the bitrate of layered video encoding is about 20% higher than ordinary encoding methods. The more layers, the more the efficiency decreases. (Currently the SDK only supports 1 base layer and 1 extension layer) Lower encoding efficiency: Under the same conditions, the encoding computational complexity of layered video encoding is higher than that of ordinary encoding methods, so the encoding efficiency is about 10% lower compared to ordinary encoding methods. Does not support hardware encoding (supports hardware decoding): Layered video encoding does not support hardware encoding, which places a greater burden on CPU performance. Fewer supported encoders, currently only the openH264 encoder supports.	When using software encoding, the performance consumption is slightly greater than layered video encoding.

Prerequisites

Before implementing the two video encoding functions, please ensure:

You have created a project in ZEGOCLOUD Console and applied for valid AppID and AppSign. For details, please refer to Console - Project Information.
You have integrated ZEGO Express SDK in the project and implemented basic audio and video streaming functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Implementation Steps

Layered Video Encoding

Enable layered video encoding

Before calling StartPublishingStream, call the SetVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDSVC to enable layered video encoding; and call the StartPublishingStream interface to start publishing stream.

ZegoVideoConfig videoConfig = new ZegoVideoConfig ();
videoConfig.codecID = ZegoVideoCodecID.SVC

engine.SetVideoConfig(videoConfig);
engine.StartPublishingStream("0012");

ZegoVideoConfig videoConfig = new ZegoVideoConfig ();
videoConfig.codecID = ZegoVideoCodecID.SVC

engine.SetVideoConfig(videoConfig);
engine.StartPublishingStream("0012");

Note

Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8" or "ZegoVideoCodecIDH265" can disable this function.

Specify the layered video to pull

After the publishing end enables layered video encoding, the playing end user can call the SetPlayStreamVideoType interface before or after playing stream, and pass in specific playing parameters to pull specific video layers. Currently supported video layers are as follows:

Enumeration Value	Description
ZegoVideoStreamTypeDefault	(Default value) Automatically select the appropriate video layer according to the network status, for example, only pull the base layer in weak networks.
ZegoVideoStreamTypeSmall	Base layer, small resolution type.
ZegoVideoStreamTypeBig	Extension layer, large resolution type.

engine.SetPlayStreamVideoType(ZegoVideoStreamType.Big, streamID)
engine.StartPlayingStream(streamID);

engine.SetPlayStreamVideoType(ZegoVideoStreamType.Big, streamID)
engine.StartPlayingStream(streamID);

Small-Large Stream Video Encoding

The implementation steps of Small-Large Stream Video Encoding are basically the same as Layered Video Encoding. The difference is that the Small-Large Stream Video Encoding function supports setting the resolution, frame rate, and bitrate of the large stream and small stream separately before publishing stream.

Enable small-large stream video encoding

Before calling StartPublishingStream, call the SetVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDH264DualStream to enable small-large stream video encoding.

ZegoVideoConfig videoConfig = new ZegoVideoConfig();
videoConfig.codecID = ZegoVideoCodecID.H264DualStream

engine.SetVideoConfig(videoConfig);

ZegoVideoConfig videoConfig = new ZegoVideoConfig();
videoConfig.codecID = ZegoVideoCodecID.H264DualStream

engine.SetVideoConfig(videoConfig);

Note

Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8" or "ZegoVideoCodecIDH265" can disable this function.

Set the parameters of the base layer and extension layer

Through the SetPublishDualStreamConfig interface, set the resolution, frame rate, and bitrate of the large stream and small stream separately, and call the StartPublishingStream interface to start publishing stream.

Warning

Must specify both the large stream and small stream parameters at the same time for the setPublishDualStreamConfig interface to take effect.
The "ratio" of the resolution of the set large stream and small stream needs to remain consistent, otherwise calling the setPublishDualStreamConfig interface will cause an error.

List<ZegoPublishDualStreamConfig> config_list = new List<ZegoPublishDualStreamConfig>();

ZegoPublishDualStreamConfig big = new ZegoPublishDualStreamConfig();
big.streamType = ZegoVideoStreamType.Big;
big.encodeHeight = 960;
big.encodeWidth = 540;
big.fps = 15;
big.bitrate = 1200;
config_list.Add(big);

ZegoPublishDualStreamConfig small = new ZegoPublishDualStreamConfig();
small.streamType = ZegoVideoStreamType.Small;
small.encodeHeight = 320;
small.encodeWidth = 180;
small.fps = 15;
small.bitrate = 300;
config_list.Add(small);
engine.SetPublishDualStreamConfig(config_list,ZegoPublishChannel.Main);

engine.StartPublishingStream("dual_streamid");

List<ZegoPublishDualStreamConfig> config_list = new List<ZegoPublishDualStreamConfig>();

ZegoPublishDualStreamConfig big = new ZegoPublishDualStreamConfig();
big.streamType = ZegoVideoStreamType.Big;
big.encodeHeight = 960;
big.encodeWidth = 540;
big.fps = 15;
big.bitrate = 1200;
config_list.Add(big);

ZegoPublishDualStreamConfig small = new ZegoPublishDualStreamConfig();
small.streamType = ZegoVideoStreamType.Small;
small.encodeHeight = 320;
small.encodeWidth = 180;
small.fps = 15;
small.bitrate = 300;
config_list.Add(small);
engine.SetPublishDualStreamConfig(config_list,ZegoPublishChannel.Main);

engine.StartPublishingStream("dual_streamid");

Specify the video stream to pull

After the publishing end enables video small-large stream encoding, the playing end user can call the SetPlayStreamVideoType interface before or after playing stream, and pass in specific playing parameters to pull specific video layers. Currently supported video layers are as follows:

Enumeration Value	Description
ZegoVideoStreamTypeDefault	(Default value) Automatically select the appropriate video layer according to the network status, for example, only pull the base layer in weak networks.
ZegoVideoStreamTypeSmall	Base layer, small resolution type.
ZegoVideoStreamTypeBig	Extension layer, large resolution type.

engine.SetPlayStreamVideoType(ZegoVideoStreamType.Big,streamID);
engine.StartPlayingStream(streamID);

engine.SetPlayStreamVideoType(ZegoVideoStreamType.Big,streamID);
engine.StartPlayingStream(streamID);

FAQ

When relaying or directly publishing to CDN, and the audience pulls streams from CDN, are layered video encoding and small-large streams effective? What are the bitrate and resolution of the streams pulled from CDN?
- Video Layered Encoding and Video Small-Large Stream Encoding use ZEGO's private protocol. The playing end can only pull video streams of different layers when pulling RTC streams or L3 streams from the ZEGO server.
- In the direct publishing to CDN scenario, since it does not go through the ZEGO server, the layering of the stream is invalid, and the SDK will fall back to H.264 encoding. The resolution and bitrate of the stream pulled from CDN are consistent with the resolution and bitrate set by the publishing user.
- In the relaying to CDN scenario, since CDN pulling does not use ZEGO's private protocol, the stream relayed by the ZEGO server to the CDN server does not support layered video encoding and video small-large stream encoding. You can only choose one from relaying the base layer or relaying the extension layer. The resolution, bitrate, and frame rate when pulling from CDN depend on whether the relayed stream is the base layer or the extension layer.

Warning

When relaying to CDN, the extension layer is relayed by default. If the business needs to relay the base layer to CDN, please contact ZEGOCLOUD Technical Support for configuration.