Video Small-Large Stream and Layered Encoding

Feature Overview

When the following requirements appear in developers' co-hosting or stream mixing business scenarios, it is recommended to use the Video Layered Encoding or Video Small-Large Stream Encoding features provided by the SDK:

• Need to display video streams of different quality on different terminals.
• Need to maintain smooth co-hosting in poor network environments.
• Need to adaptively pull the quality of video streams based on network status.

Video Layered Encoding (H.264 SVC) is an extension of H.264 AVC, supporting bitstreams with layered characteristics. Video Small-Large Stream Encoding (H.264 DualStream) refers to the concept of H.264 SVC. By layering video bitstreams, it guarantees a smooth experience for terminals with different networks and device performance.

Both encoding methods divide the bitstream into "base layer" and "extension layer", which can provide better experience for users with different network states and device performance. The base layer guarantees the most basic video quality, while the extension layer is a supplement to the base layer. For users with better networks, they can pull the extension layer to get a better experience. For users with poor network states, only pulling the base layer can guarantee basic video quality. The following is a detailed comparison of the two functions:

Example Source Code Download

Please refer to Download Example Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/AdvancedVideoProcessing/src/main/java/im/zego/advancedvideoprocessing/encodinganddecoding" directory.

Prerequisites

Before implementing both video encoding functions, please ensure:

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
• You have integrated ZEGO Express SDK in your project and implemented basic audio and video streaming functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Implementation Steps

Layered Video Encoding

Enable layered video encoding

Before calling startPublishingStream, call the setVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDSVC to enable layered video encoding; and call the startPublishingStream interface to start publishing stream.

ZegoVideoConfig videoConfig = new ZegoVideoConfig ();
videoConfig.codecID = ZegoVideoCodecID.ZegoVideoCodecIDSVC;

engine.setVideoConfig(videoConfig);
engine.startPublishingStream("0012");

ZegoVideoConfig videoConfig = new ZegoVideoConfig ();
videoConfig.codecID = ZegoVideoCodecID.ZegoVideoCodecIDSVC;

engine.setVideoConfig(videoConfig);
engine.startPublishingStream("0012");

Specify the layered video to pull

After the publishing end enables layered video encoding, the play stream end users can call the setPlayStreamVideoType interface before or after playing stream to pull specific video layers by passing in specific play stream parameters. The currently supported video layers are as follows:

engine.setPlayStreamVideoType(ZegoVideoStreamTypeBig, streamID);
engine.startPlayingStream(streamID, zegoCanvas);

engine.setPlayStreamVideoType(ZegoVideoStreamTypeBig, streamID);
engine.startPlayingStream(streamID, zegoCanvas);

Small-Large Stream Video Encoding

The implementation steps of Small-Large Stream Video Encoding and Layered Video Encoding are basically the same. The difference is that the Small-Large Video Stream Encoding function supports setting the resolution, frame rate, and bitrate of the large stream and small stream respectively before publishing stream.

Enable small-large stream video encoding

Before calling startPublishingStream, call the setVideoConfig interface to set the parameter codecID in the ZegoVideoConfig class to ZegoVideoCodecIDH264DualStream to enable small-large stream video encoding; and call the startPublishingStream interface to start publishing stream.

ZegoVideoConfig videoConfig = new ZegoVideoConfig ();
videoConfig.codecID = ZegoVideoCodecID.H264_DUAL_STREAM;

engine.setVideoConfig(videoConfig);

ZegoVideoConfig videoConfig = new ZegoVideoConfig ();
videoConfig.codecID = ZegoVideoCodecID.H264_DUAL_STREAM;

engine.setVideoConfig(videoConfig);

Set parameters for base layer and extension layer

Through the setPublishDualStreamConfig interface, set the resolution, frame rate, and bitrate of the large stream and small stream respectively, and call the startPublishingStream interface to start publishing stream.

ArrayList<ZegoPublishDualStreamConfig> config_list = new ArrayList<ZegoPublishDualStreamConfig>();

ZegoPublishDualStreamConfig big = new ZegoPublishDualStreamConfig();
big.streamType = ZegoVideoStreamType.BIG;
big.encodeHeight = 960;
big.encodeWidth = 540;
big.fps = 15;
big.bitrate = 1200;
config_list.add(big);

ZegoPublishDualStreamConfig small = new ZegoPublishDualStreamConfig();
small.streamType = ZegoVideoStreamType.SMALL;
small.encodeHeight = 320;
small.encodeWidth = 180;
small.fps = 15;
small.bitrate = 300;
config_list.add(small);
engine.setPublishDualStreamConfig(config_list,ZegoPublishChannel.MAIN);

engine.startPublishingStream("dual_streamid");

ArrayList<ZegoPublishDualStreamConfig> config_list = new ArrayList<ZegoPublishDualStreamConfig>();

ZegoPublishDualStreamConfig big = new ZegoPublishDualStreamConfig();
big.streamType = ZegoVideoStreamType.BIG;
big.encodeHeight = 960;
big.encodeWidth = 540;
big.fps = 15;
big.bitrate = 1200;
config_list.add(big);

ZegoPublishDualStreamConfig small = new ZegoPublishDualStreamConfig();
small.streamType = ZegoVideoStreamType.SMALL;
small.encodeHeight = 320;
small.encodeWidth = 180;
small.fps = 15;
small.bitrate = 300;
config_list.add(small);
engine.setPublishDualStreamConfig(config_list,ZegoPublishChannel.MAIN);

engine.startPublishingStream("dual_streamid");

Specify the video stream to pull

After the publishing end enables video small-large stream encoding, the play stream end users can call the setPlayStreamVideoType interface before or after playing stream to pull specific video layers by passing in specific play stream parameters. The currently supported video layers are as follows:

engine.setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
engine.startPlayingStream(streamID,canvans);

engine.setPlayStreamVideoType(ZegoVideoStreamTypeBig,streamID);
engine.startPlayingStream(streamID,canvans);

FAQ

1. For forwarding or direct pushing to CDN, the audience pulls streams from CDN. Are layered video encoding and small-large streams effective? What is the bitrate and resolution of the streams pulled from CDN?

   • Video Layered Encoding and Video Small-Large Stream Encoding use ZEGO's private protocol. The play stream end can only pull different layered video streams when pulling RTC streams or L3 streams from ZEGO servers.

   • In the direct push CDN scenario, since it does not go through ZEGO servers, the layering of the bitstream is invalid, and the SDK will fall back to H.264 encoding. The resolution and bitrate of the stream pulled from CDN are consistent with the resolution and bitrate set by the publishing user.

   • In the forwarding CDN scenario, since CDN play stream does not use ZEGO's private protocol, the stream forwarded by ZEGO servers to CDN servers does not support layered video encoding and video small-large stream encoding. You can only choose one between forwarding the base layer or forwarding the extension layer. The resolution, bitrate, and frame rate when pulling from CDN depend on whether the forwarding is the base layer or the extension layer.

     Warning

     When forwarding to CDN, the extension layer is forwarded by default. If the business needs to forward the base layer to CDN, please contact ZEGO Technical Support for configuration.