Small/Large Video Stream and Layered Encoding
Feature Overview
When the following requirements arise in your video call or stream mixing business scenarios, it is recommended to use the Video Layered Encoding or Video Small/Large Stream Encoding features provided by the SDK:
- Need to display video streams of different quality on different terminals.
- Need to maintain smooth video calls in poor network environments.
- Need to adaptively pull the quality of video streams based on network status.
Video Layered Encoding (H.264 SVC) is an extension of H.264 AVC that supports bitstreams with layered characteristics. Video Small/Large Stream Encoding (H.264 DualStream) adopts the concept of H.264 SVC and ensures a smooth experience for terminals with different network and device performance by layering video bitstreams.
Both encoding methods divide the bitstream into "Base Layer" and "Extension Layer", providing a better experience for users with different network statuses and device performance. The base layer guarantees the most basic video quality, while the extension layer supplements the base layer. Users with good networks can pull the extension layer for a better experience, while users with poor network status can only pull the base layer to ensure basic video quality. The following is a detailed comparison of the two features:
| - | Video Layered Encoding (H.264 SVC) | Video Small/Large Stream Encoding (H.264 DualStream) |
|---|---|---|
| Implementation Logic | Start 1 encoder to encode bitstreams with different parameters as the base layer and extension layer of the bitstream. | Start 2 encoders to encode bitstreams with different parameters as the base layer and extension layer of the bitstream. |
| Protocol Used | Uses ZEGO's private protocol. The stream playing end can only pull different layered video streams from ZEGO servers. | |
| Encoding/Decoding | The encoding process of the base layer and extension layer is not independent, but can be decoded independently. | The encoding and decoding processes of the base layer and extension layer are both independent of each other. |
| Publishing Stream | Select the encoding format as "ZegoVideoCodecIDSvc". |
|
| Playing Stream | There is no difference in interface calls. The same user can only pull one layer of the video stream at the same time. By default, when the network is good, only the extension layer is pulled; when the network is poor, only the base layer is pulled. | |
| The resolution of the pulled base layer is 50% of the extension layer, the bitrate is 25% of the extension layer, and the frame rate remains the same. It cannot be configured customly. | Depends on the video parameters of the base layer and extension layer set by the publishing end. | |
| Advantages |
|
|
| Disadvantages |
| When using software encoding, performance consumption is slightly higher than layered video encoding. |
Download Example Source Code
Please refer to Download Example Source Code to get the source code.
For related source code, please check files in the "/ZegoExpressExample/Examples/AdvancedVideoProcessing/EncodingAndDecoding" directory.
Prerequisites
Before implementing both video encoding features, please ensure:
- You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
- You have integrated ZEGO Express SDK in the project and implemented basic audio and video publishing and playing functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation Flow.
Implementation Steps
Layered Video Encoding
Enable Layered Video Encoding
Before calling startPublishingStream, call the setVideoConfig interface and set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDSVC to enable layered video encoding; and call the startPublishingStream interface to start publishing stream.
ZegoVideoConfig videoConfig;
videoConfig.codecID = ZEGO_VIDEO_CODEC_ID_SVC;
engine->setVideoConfig(videoConfig);
std::string streamID = "MultiLayer-1";
engine->startPublishingStream(streamID);Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8", or "ZegoVideoCodecIDH265" can disable this feature.
Specify Layered Video to Play
After the publishing end enables layered video encoding, the playing end user can call the setPlayStreamVideoType interface before or after playing stream, passing in specific playing parameters to pull a specific video layer. Currently supported video layers are as follows:
| Enum Value | Description |
|---|---|
| ZegoVideoStreamTypeDefault | （Default value）Automatically selects the appropriate video layer based on network status, e.g., only pull the base layer in weak networks. |
| ZegoVideoStreamTypeSmall | Base layer, small resolution type. |
| ZegoVideoStreamTypeBig | Extension layer, large resolution type. |
ZegoExpressSDK::getEngine()->setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
ZegoExpressSDK::getEngine()->startPlayingStream(streamID, canvas:playCanvas);Small/Large Stream Video Encoding
The implementation steps of Small/Large Stream Video Encoding are basically the same as Layered Video Encoding, except that Small/Large Stream Video Encoding supports setting the resolution, frame rate, and bitrate of the large stream and small stream separately before publishing stream.
Enable Small/Large Stream Video Encoding
Before calling startPublishingStream, call the setVideoConfig interface and set the parameter codecID in the ZegoVideoConfig class to ZEGO_VIDEO_CODEC_ID_H264_DUAL_STREAM to enable small/large stream video encoding; and call the startPublishingStream interface to start publishing stream.
ZegoVideoConfig videoConfig;
videoConfig.codecID = ZEGO_VIDEO_CODEC_ID_H264_DUAL_STREAM;
ZegoExpressSDK::getEngine()->setVideoConfig(video_config);Setting "codecID" to "ZegoVideoCodecIDDefault", "ZegoVideoCodecIDVP8", or "ZegoVideoCodecIDH265" can disable this feature.
Set Base Layer and Extension Layer Parameters
Through the setPublishDualStreamConfig interface, set the resolution, frame rate, and bitrate of the large stream and small stream separately, and call the startPublishingStream interface to start publishing stream.
- Must specify both the large stream and small stream parameters for the setPublishDualStreamConfig interface to take effect.
- The "ratio" of the resolution set for the large stream and small stream must be consistent, otherwise calling the setPublishDualStreamConfig interface will result in an error.
std::vector<ZegoPublishDualStreamConfig> dual_stream_config;
ZegoPublishDualStreamConfig stream_config_big;
stream_config_big.streamType = ZEGO_VIDEO_STREAM_TYPE_BIG;
stream_config_big.encodeWidth = 960;
stream_config_big.encodeHeight = 540;
stream_config_big.fps = 15;
stream_config_big.bitrate = 1200;
dual_stream_config.push_back(stream_config_big);
ZegoPublishDualStreamConfig stream_config_small;
stream_config_small.streamType = ZEGO_VIDEO_STREAM_TYPE_SMALL;
stream_config_small.encodeWidth = 320;
stream_config_small.encodeHeight = 180;
stream_config_small.fps = 15;
stream_config_small.bitrate = 300;
dual_stream_config.push_back(stream_config_small);
ZegoExpressSDK::getEngine()->setPublishDualStreamConfig(dual_stream_config, ZegoPublishChannel::ZEGO_PUBLISH_CHANNEL_MAIN);
std::string streamid = "dual_stream";
ZegoExpressSDK::getEngine()->startPublishingStream(streamid);Specify Video Stream to Play
After the publishing end enables video small/large stream encoding, the playing end user can call the setPlayStreamVideoType interface before or after playing stream, passing in specific playing parameters to pull a specific video layer. Currently supported video layers are as follows:
| Enum Value | Description |
|---|---|
| ZegoVideoStreamTypeDefault | （Default value）Automatically selects the appropriate video layer based on network status, e.g., only pull the base layer in weak networks. |
| ZegoVideoStreamTypeSmall | Base layer, small resolution type. |
| ZegoVideoStreamTypeBig | Extension layer, large resolution type. |
ZegoExpressSDK::getEngine()->setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
ZegoExpressSDK::getEngine()->startPlayingStream(streamID, canvas:playCanvas);FAQ
-
When relaying or directly publishing to CDN, viewers play stream from CDN. Are layered video encoding and small/large streams effective? What is the bitrate and resolution of the stream pulled from CDN?
-
Video Layered EncodingandVideo Small/Large Stream Encodinguse ZEGO's private protocol. The stream playing end can only pull different layered video streams when pulling RTC streams or L3 streams from ZEGO servers. -
In the direct CDN publishing scenario, since it does not go through ZEGO servers, the layering of the bitstream is invalid, and the SDK will fall back to H.264 encoding. The resolution and bitrate of the stream pulled from CDN are consistent with the resolution and bitrate set by the publishing user.
-
In the relay to CDN scenario, since CDN stream playing does not use ZEGO's private protocol, the stream relayed by ZEGO servers to CDN servers does not support layered video encoding or video small/large stream encoding. You can only choose one between relaying the base layer or relaying the extension layer. The resolution, bitrate, and frame rate when playing from CDN depend on whether the base layer or extension layer is relayed.
-
When relaying to CDN, the extension layer is relayed by default. If your business needs to relay the base layer to CDN, please contact ZEGOCLOUD Technical Support for configuration.