Video Small/Large Stream and Layered Encoding

2024-04-01

Feature Overview

When the following requirements appear in the developer's co-hosting or stream mixing business scenarios, it is recommended to use the SDK's Video Layered Encoding or Video Small/Large Stream Encoding feature:

Need different terminals to display video streams of different quality.
Need to maintain smooth co-hosting in poor network environments.
Need to adaptively pull video stream quality based on network status.

Video Layered Encoding (H.264 SVC) is an extension of H.264 AVC, supporting bitstreams with layered characteristics. Video Small/Large Stream Encoding (H.264 DualStream) draws on the concept of H.264 SVC, ensuring a smooth experience for terminals with different network and device performance by layering video bitstreams.

Both encoding methods divide the bitstream into "base layer" and "enhancement layer", which can provide better experience for users with different network states and device performance. The base layer guarantees the most basic video quality, while the enhancement layer supplements the base layer. For users with better networks, the enhancement layer can be pulled for better experience; for users with poor network states, only pulling the base layer can guarantee basic video quality. The following is a detailed comparison of the two features:

-	Video Layered Encoding (H.264 SVC)	Video Small/Large Stream Encoding (H.264 DualStream)
Implementation Logic	Start 1 encoder to encode bitstreams with different parameters as the base layer and enhancement layer of the bitstream.	Start 2 encoders to encode bitstreams with different parameters as the base layer and enhancement layer of the bitstream.
Protocol Used	Use ZEGO's private protocol. The playing side can only pull video streams of different layers from ZEGO servers.
Encoding/Decoding	The encoding process of base layer and enhancement layer is not independent, but can be decoded independently.	The encoding and decoding processes of base layer and enhancement layer are independent of each other.
Publishing	Select encoding format as "ZegoVideoCodecIDSvc".	Select encoding format as "ZegoVideoCodecIDH264DualStream". (Optional) Configure video parameters for base layer and enhancement layer respectively through the [setPublishDualStreamConfig] interface.
Playing	There is no difference in interface calls. The same user can only pull one layer of the video stream at the same time. By default, when the network is good, only the enhancement layer is pulled; when the network is poor, only the base layer is pulled.
Playing	The resolution of the pulled base layer is 50% of the enhancement layer, the bitrate is 25% of the enhancement layer, and the frame rate remains the same, which cannot be customized.	Depends on the video parameters of the base layer and enhancement layer set by the publishing side.
Advantages	Can generate different bitstreams or extract different bitstreams as needed. Using layered video encoding to encode once is more efficient than using ordinary encoding methods to encode multiple times. More flexible application. Stronger network adaptability.	Can separately configure video parameters for enhancement layer and base layer. Can use hardware encoding to reduce CPU performance burden. Better scalability, supporting multiple encoding standards in the future. Better universality, mainstream software and hardware encoders support it.
Disadvantages	Slightly lower compression efficiency: Under the same conditions, layered video encoding is about 20% less efficient than ordinary encoding methods, that is, to achieve the same video quality as ordinary encoding methods, the bitrate of layered video encoding is about 20% higher than ordinary encoding methods. The more layers, the more the efficiency decreases. (Currently, the SDK only supports 1 base layer and 1 enhancement layer) Low encoding efficiency: Under the same conditions, layered video encoding has higher encoding computational complexity than ordinary encoding methods, so the encoding efficiency is about 10% lower than ordinary encoding methods. Does not support hardware encoding (supports hardware decoding): Layered video encoding does not support hardware encoding, which has a large burden on CPU performance. Fewer supported encoders. Currently, only the openH264 encoder supports it.	When using software encoding, performance consumption is slightly greater than layered video encoding.

Download Sample Source Code

Please refer to Download Sample Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Examples/AdvancedVideoProcessing/EncodingAndDecoding" directory.

Prerequisites

Before implementing the two video encoding functions, ensure that:

A project has been created in the ZEGOCLOUD Console, and a valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.
ZEGO Express SDK has been integrated into the project, and basic audio/video stream publishing and playing functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Implementation Steps

Layered Video Encoding

Enable layered video encoding

Before calling startPublishingStream, call the setVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDSVC to enable layered video encoding; and call the startPublishingStream interface to start publishing stream.

ZegoVideoConfig videoConfig;
videoConfig.codecID = ZEGO_VIDEO_CODEC_ID_SVC;
engine->setVideoConfig(videoConfig);

std::string streamID = "MultiLayer-1";
engine->startPublishingStream(streamID);

ZegoVideoConfig videoConfig;
videoConfig.codecID = ZEGO_VIDEO_CODEC_ID_SVC;
engine->setVideoConfig(videoConfig);

std::string streamID = "MultiLayer-1";
engine->startPublishingStream(streamID);

Note

Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8" or "ZegoVideoCodecIDH265" can disable this feature.

Specify the layered video to pull

After the publishing side enables layered video encoding, users on the playing side can call the setPlayStreamVideoType interface before or after playing stream, passing in specific playing parameters to pull specific video layers. The currently supported video layers are as follows:

Enumeration Value	Description
ZegoVideoStreamTypeDefault	(Default) Automatically select the appropriate video layer based on network status, for example, only pull the base layer in weak network conditions.
ZegoVideoStreamTypeSmall	Base layer, small resolution type.
ZegoVideoStreamTypeBig	Enhancement layer, large resolution type.

ZegoExpressSDK::getEngine()->setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
ZegoExpressSDK::getEngine()->startPlayingStream(streamID, canvas:playCanvas);

ZegoExpressSDK::getEngine()->setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
ZegoExpressSDK::getEngine()->startPlayingStream(streamID, canvas:playCanvas);

Small/Large Stream Video Encoding

The implementation steps of Small/Large Stream Video Encoding are basically the same as Layered Video Encoding, except that Small/Large Video Stream Encoding supports setting the resolution, frame rate, and bitrate of the large stream and small stream separately before publishing stream.

Enable small/large stream video encoding

Before calling startPublishingStream, call the setVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZEGO_VIDEO_CODEC_ID_H264_DUAL_STREAM to enable small/large stream video encoding; and call the startPublishingStream interface to start publishing stream.

ZegoVideoConfig videoConfig;
videoConfig.codecID = ZEGO_VIDEO_CODEC_ID_H264_DUAL_STREAM;
ZegoExpressSDK::getEngine()->setVideoConfig(video_config);

ZegoVideoConfig videoConfig;
videoConfig.codecID = ZEGO_VIDEO_CODEC_ID_H264_DUAL_STREAM;
ZegoExpressSDK::getEngine()->setVideoConfig(video_config);

Note

Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8" or "ZegoVideoCodecIDH265" can disable this feature.

Set base layer and enhancement layer parameters

Through the setPublishDualStreamConfig interface, set the resolution, frame rate, and bitrate of the large stream and small stream separately, and call the startPublishingStream interface to start publishing stream.

Warning

Must specify both large stream and small stream parameters at the same time for the setPublishDualStreamConfig interface to take effect.
The "ratio" of the resolutions set for the large stream and small stream needs to be consistent, otherwise calling the setPublishDualStreamConfig interface will result in an error.

std::vector<ZegoPublishDualStreamConfig> dual_stream_config;
ZegoPublishDualStreamConfig stream_config_big;
stream_config_big.streamType = ZEGO_VIDEO_STREAM_TYPE_BIG;
stream_config_big.encodeWidth = 960;
stream_config_big.encodeHeight = 540;
stream_config_big.fps = 15;
stream_config_big.bitrate = 1200;
dual_stream_config.push_back(stream_config_big);

ZegoPublishDualStreamConfig stream_config_small;
stream_config_small.streamType = ZEGO_VIDEO_STREAM_TYPE_SMALL;
stream_config_small.encodeWidth = 320;
stream_config_small.encodeHeight = 180;
stream_config_small.fps = 15;
stream_config_small.bitrate = 300;
dual_stream_config.push_back(stream_config_small);
ZegoExpressSDK::getEngine()->setPublishDualStreamConfig(dual_stream_config, ZegoPublishChannel::ZEGO_PUBLISH_CHANNEL_MAIN);

std::string streamid = "dual_stream";
ZegoExpressSDK::getEngine()->startPublishingStream(streamid);

std::vector<ZegoPublishDualStreamConfig> dual_stream_config;
ZegoPublishDualStreamConfig stream_config_big;
stream_config_big.streamType = ZEGO_VIDEO_STREAM_TYPE_BIG;
stream_config_big.encodeWidth = 960;
stream_config_big.encodeHeight = 540;
stream_config_big.fps = 15;
stream_config_big.bitrate = 1200;
dual_stream_config.push_back(stream_config_big);

ZegoPublishDualStreamConfig stream_config_small;
stream_config_small.streamType = ZEGO_VIDEO_STREAM_TYPE_SMALL;
stream_config_small.encodeWidth = 320;
stream_config_small.encodeHeight = 180;
stream_config_small.fps = 15;
stream_config_small.bitrate = 300;
dual_stream_config.push_back(stream_config_small);
ZegoExpressSDK::getEngine()->setPublishDualStreamConfig(dual_stream_config, ZegoPublishChannel::ZEGO_PUBLISH_CHANNEL_MAIN);

std::string streamid = "dual_stream";
ZegoExpressSDK::getEngine()->startPublishingStream(streamid);

Specify the video stream to pull

After the publishing side enables video small/large stream encoding, users on the playing side can call the setPlayStreamVideoType interface before or after playing stream, passing in specific playing parameters to pull specific video layers. The currently supported video layers are as follows:

Enumeration Value	Description
ZegoVideoStreamTypeDefault	(Default) Automatically select the appropriate video layer based on network status, for example, only pull the base layer in weak network conditions.
ZegoVideoStreamTypeSmall	Base layer, small resolution type.
ZegoVideoStreamTypeBig	Enhancement layer, large resolution type.

ZegoExpressSDK::getEngine()->setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
ZegoExpressSDK::getEngine()->startPlayingStream(streamID, canvas:playCanvas);

ZegoExpressSDK::getEngine()->setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
ZegoExpressSDK::getEngine()->startPlayingStream(streamID, canvas:playCanvas);

FAQ

For relaying or directly publishing to CDN, when the audience pulls streams from CDN, are layered video encoding and small/large stream effective? What are the bitrate and resolution of streams pulled from CDN?
- Video Layered Encoding and Video Small/Large Stream Encoding use ZEGO's private protocol. The playing side can only pull video streams of different layers when pulling RTC streams or L3 streams from ZEGO servers.
- In the direct publishing to CDN scenario, since it does not go through ZEGO servers, the bitstream layering is invalid, and the SDK will fall back to H.264 encoding. The resolution and bitrate of streams pulled from CDN are consistent with the resolution and bitrate set by the publishing user.
- In the relaying to CDN scenario, since CDN playing does not use ZEGO's private protocol, the stream relayed by ZEGO servers to CDN servers does not support layered video encoding or video small/large stream encoding. Only one of relaying base layer or relaying enhancement layer can be selected. The resolution, bitrate, and frame rate when pulling from CDN depend on whether the relayed base layer or enhancement layer is relayed.
  
  Warning
  When relaying to CDN, the enhancement layer is relayed by default. If the business needs to relay the base layer to CDN, please contact ZEGOCLOUD Technical Support for configuration.