Video Large/Small Stream and Layered Encoding

2024-04-01

Feature Introduction

When the following requirements appear in developers' co-hosting or stream mixing business scenarios, it is recommended to use the Video Layered Encoding or Video Large/Small Stream Encoding features provided by the SDK:

Need different terminals to display video streams of different quality.
Need to maintain smooth co-hosting in poor network environments.
Need to adaptively pull the quality of video streams based on network status.

Video Layered Encoding (H.264 SVC) is an extension of H.264 AVC, supporting bitstreams with layered characteristics. Video Large/Small Stream Encoding (H.264 DualStream) references the concept of H.264 SVC, ensuring smooth experience for terminals with different network and device performance through video stream layering.

Both encoding methods divide the bitstream into a "base layer" and an "enhancement layer", providing better experience for users with different network states and device performance. The base layer guarantees the most basic video quality, while the enhancement layer is a supplement to the base layer. For users with better networks, they can pull the enhancement layer for better experience; for users with poor network states, pulling only the base layer can guarantee basic video quality. The following is a detailed comparison of the two features:

-	Video Layered Encoding (H.264 SVC)	Video Large/Small Stream Encoding (H.264 DualStream)
Implementation Logic	Start 1 encoder to encode bitstreams with different parameters separately, as the base layer and enhancement layer of the bitstream.	Start 2 encoders to encode bitstreams with different parameters separately, as the base layer and enhancement layer of the bitstream.
Protocol Used	Use ZEGO's private protocol. The playing end can only pull different layered video streams from ZEGO servers.
Encoding/Decoding	The encoding process of base layer and enhancement layer is not independent, but can be decoded independently.	The encoding and decoding processes of base layer and enhancement layer are both independent of each other.
Publishing Stream	Select encoding format as "ZegoVideoCodecIDSvc".	Select encoding format as "ZegoVideoCodecIDH264DualStream". (Optional) Configure video parameters for base layer and enhancement layer separately through the [setPublishDualStreamConfig] interface.
Playing Stream	There is no difference in interface calls. The same user can only pull one layer of the video stream at the same time. By default, when the network is good, only the enhancement layer is pulled; when the network is poor, only the base layer is pulled.
Playing Stream	The resolution of the pulled base layer is 50% of the enhancement layer, the bitrate is 25% of the enhancement layer, and the frame rate remains consistent, which cannot be customized.	Depends on the video parameters of the base layer and enhancement layer set by the publishing end.
Advantages	Can generate different bitstreams or extract different bitstreams as needed. Using layered video encoding to encode once is more efficient than using ordinary encoding methods to encode multiple times. More flexible application. Stronger network adaptability.	Video parameters for enhancement layer and base layer can be configured separately. Can use hardware encoding to reduce CPU performance burden. Better scalability, can support multiple encoding standards in the future. Better universality, mainstream software and hardware encoders support it.
Disadvantages	Slightly lower compression efficiency: Under the same conditions, layered video encoding is about 20% less efficient than ordinary encoding methods. That is, to achieve the same video quality as ordinary encoding methods, the bitrate of layered video encoding is 20% higher than ordinary encoding methods. The more layers, the more the efficiency decreases. (Currently the SDK only supports 1 base layer and 1 enhancement layer) Lower encoding efficiency: Under the same conditions, layered video encoding has higher encoding computational complexity than ordinary encoding methods, so encoding efficiency is about 10% lower than ordinary encoding methods. Does not support hardware encoding (supports hardware decoding): Layered video encoding does not support hardware encoding, which places a greater burden on CPU performance. Fewer supported encoders, currently only the openH264 encoder supports it.	When using software encoding, performance consumption is slightly higher than layered video encoding.

Sample Source Code Download

Please refer to Download Sample Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Examples/AdvancedVideoProcessing/Encoding&Decoding" directory.

Prerequisites

Before implementing both video encoding features, please ensure:

You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
You have integrated ZEGO Express SDK in the project and implemented basic audio and video stream publishing and playing functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation Flow.

Implementation Steps

Layered Video Encoding

Enable layered video encoding

Before calling startPublishingStream, call the setVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDSVC to enable layered video encoding; and call the startPublishingStream interface to start publishing stream.

ZegoVideoConfig *videoConfig = [[ZegoVideoConfig alloc] init];
videoConfig.codecID = ZegoVideoCodecIDSVC;
[[ZegoExpressEngine sharedEngine] setVideoConfig:videoConfig];

self.streamID = @"0012";
[[ZegoExpressEngine sharedEngine] startPublishingStream:self.streamID];

ZegoVideoConfig *videoConfig = [[ZegoVideoConfig alloc] init];
videoConfig.codecID = ZegoVideoCodecIDSVC;
[[ZegoExpressEngine sharedEngine] setVideoConfig:videoConfig];

self.streamID = @"0012";
[[ZegoExpressEngine sharedEngine] startPublishingStream:self.streamID];

Note

Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8" or "ZegoVideoCodecIDH265" can disable this feature.

Specify the layered video to pull

After the publishing end enables layered video encoding, users at the playing end can call the setPlayStreamVideoType interface before or after playing stream, passing in specific playing parameters to pull specific video layers. Currently supported video layers are as follows:

Enumeration Value	Description
ZegoVideoStreamTypeDefault	(Default) Automatically select the appropriate video layer based on network status, for example, only pull the base layer in weak networks.
ZegoVideoStreamTypeSmall	Base layer, small resolution type.
ZegoVideoStreamTypeBig	Enhancement layer, large resolution type.

[[ZegoExpressEngine sharedEngine] setPlayStreamVideoType:ZegoVideoStreamTypeBig streamID:self.streamID];
[[ZegoExpressEngine sharedEngine] startPlayingStream:self.streamID canvas:playCanvas];

[[ZegoExpressEngine sharedEngine] setPlayStreamVideoType:ZegoVideoStreamTypeBig streamID:self.streamID];
[[ZegoExpressEngine sharedEngine] startPlayingStream:self.streamID canvas:playCanvas];

Large/Small Stream Video Encoding

The implementation steps of Large/Small Stream Video Encoding are basically the same as Layered Video Encoding, except that Large/Small Stream Video Encoding supports setting the resolution, frame rate, and bitrate of the large stream and small stream separately before publishing stream.

Enable large/small stream video encoding

Before calling startPublishingStream, call the setVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDH264DualStream to enable large/small stream video encoding; and call the startPublishingStream interface to start publishing stream.

ZegoVideoConfig *videoConfig = [[ZegoVideoConfig alloc] init];
videoConfig.codecID = ZegoVideoCodecIDH264DualStream;
[[ZegoExpressEngine sharedEngine] setVideoConfig:videoConfig];

ZegoVideoConfig *videoConfig = [[ZegoVideoConfig alloc] init];
videoConfig.codecID = ZegoVideoCodecIDH264DualStream;
[[ZegoExpressEngine sharedEngine] setVideoConfig:videoConfig];

Note

Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8" or "ZegoVideoCodecIDH265" can disable this feature.

Set parameters for base layer and enhancement layer

Through the setPublishDualStreamConfig interface, set the resolution, frame rate, and bitrate of the large stream and small stream separately, and call the startPublishingStream interface to start publishing stream.

Warning

Must specify parameters for both large stream and small stream at the same time for the setPublishDualStreamConfig interface to take effect.
The "ratio" of the resolution set for the large stream and small stream needs to remain consistent, otherwise calling the setPublishDualStreamConfig interface will result in an error.

NSMutableArray<ZegoPublishDualStreamConfig*>* dualStreamConfig = [[NSMutableArray alloc] initWithCapacity:2];
ZegoPublishDualStreamConfig* config_big = [[ZegoPublishDualStreamConfig alloc] init];
config_big.streamType = ZegoVideoStreamTypeBig;
config_big.fps = 15;
config_big.bitrate = 1200;
config_big.encodeResolution = CGSizeMake(540,960);

[dualStreamConfig addObject:config_big];

ZegoPublishDualStreamConfig* config_small = [[ZegoPublishDualStreamConfig alloc] init];
config_small.streamType = ZegoVideoStreamTypeSmall;
config_small.fps = 15;
config_small.bitrate = 300;
config_small.encodeResolution = CGSizeMake(180,320);

[dualStreamConfig addObject:config_small];

[ZegoExpressEngine.sharedEngine setPublishDualStreamConfig:dualStreamConfig channel:ZegoPublishChannelMain];

self.streamID = @"0012";
[[ZegoExpressEngine sharedEngine] startPublishingStream:self.streamID];

NSMutableArray<ZegoPublishDualStreamConfig*>* dualStreamConfig = [[NSMutableArray alloc] initWithCapacity:2];
ZegoPublishDualStreamConfig* config_big = [[ZegoPublishDualStreamConfig alloc] init];
config_big.streamType = ZegoVideoStreamTypeBig;
config_big.fps = 15;
config_big.bitrate = 1200;
config_big.encodeResolution = CGSizeMake(540,960);

[dualStreamConfig addObject:config_big];

ZegoPublishDualStreamConfig* config_small = [[ZegoPublishDualStreamConfig alloc] init];
config_small.streamType = ZegoVideoStreamTypeSmall;
config_small.fps = 15;
config_small.bitrate = 300;
config_small.encodeResolution = CGSizeMake(180,320);

[dualStreamConfig addObject:config_small];

[ZegoExpressEngine.sharedEngine setPublishDualStreamConfig:dualStreamConfig channel:ZegoPublishChannelMain];

self.streamID = @"0012";
[[ZegoExpressEngine sharedEngine] startPublishingStream:self.streamID];

Specify the video stream to pull

After the publishing end enables video large/small stream encoding, users at the playing end can call the setPlayStreamVideoType interface before or after playing stream, passing in specific playing parameters to pull specific video layers. Currently supported video layers are as follows:

Enumeration Value	Description
ZegoVideoStreamTypeDefault	(Default) Automatically select the appropriate video layer based on network status, for example, only pull the base layer in weak networks.
ZegoVideoStreamTypeSmall	Base layer, small resolution type.
ZegoVideoStreamTypeBig	Enhancement layer, large resolution type.

[[ZegoExpressEngine sharedEngine] setPlayStreamVideoType:ZegoVideoStreamTypeBig streamID:self.streamID];
[[ZegoExpressEngine sharedEngine] startPlayingStream:self.streamID canvas:playCanvas];

[[ZegoExpressEngine sharedEngine] setPlayStreamVideoType:ZegoVideoStreamTypeBig streamID:self.streamID];
[[ZegoExpressEngine sharedEngine] startPlayingStream:self.streamID canvas:playCanvas];

FAQ

For relaying to or directly publishing to CDN, when viewers pull streams from CDN, are layered video encoding and large/small streams effective? What are the bitrate and resolution of streams pulled from CDN?
- Video Layered Encoding and Video Large/Small Stream Encoding use ZEGO's private protocol. The playing end can only pull different layered video streams when pulling RTC streams or L3 streams from ZEGO servers.
- In the direct CDN publishing scenario, since it does not go through ZEGO servers, the layering of the bitstream is invalid, and the SDK will fall back to H.264 encoding. The resolution and bitrate of streams pulled from CDN are consistent with the resolution and bitrate set by the publishing user.
- In the relay to CDN scenario, since CDN pulling does not use ZEGO's private protocol, the stream relayed by ZEGO servers to CDN servers does not support layered video encoding and video large/small stream encoding. You can only choose one from relaying the base layer or relaying the enhancement layer. The resolution, bitrate, and frame rate when pulling from CDN depend on whether the relayed layer is the base layer or the enhancement layer.
  
  Warning
  When relaying to CDN, the enhancement layer is relayed by default. If the business needs to relay the base layer to CDN, please contact ZEGOCLOUD Technical Support for configuration.