Supplemental Enhancement Information (SEI)
Function Overview
In audio and video streaming applications, in addition to pushing and playing audio and video content through the streaming channel, you can also use streaming SEI (Supplemental Enhancement Information) to package text information with audio and video content through the streaming channel, publish from the host end (publishing end), and receive at the audience end (playing end), thereby achieving precise synchronization between text data and audio and video content.
Generally can be used for application scenarios such as precise layout of video screens, remote lyric synchronization, live streaming quiz, etc.
For SEI related concepts and principles, please refer to How to understand and use SEI (Supplemental Enhancement Information).
Prerequisites
Before implementing the SEI function, please ensure:
- You have created a project in ZEGOCLOUD Console and applied for valid AppID and AppSign. For details, please refer to Console - Project Information.
- You have integrated ZEGO Express SDK in the project and implemented basic audio and video streaming functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
Usage Steps
The function of sending and receiving SEI information requires sending SEI information at the publishing end and receiving SEI information at the playing end, as shown in the following figure:
1 (Only applicable to Web projects) Set SEI type
Since the SDK uses ZEGO's self-defined SEI (nalu type = 6, payload type = 243) type packaging by default, and this type is not specified in the SEI standard, it does not conflict with SEI in video encoders or video files. However, when developers need to use third-party decoders to decode (such as FFmpeg), it will result in the inability to decode the correct SEI. At this time, it is necessary to call the setSEIConfig interface before publishing stream to change the SEI type sent by the SDK, using the UserUnregister SEI (nalu type = 6, payload type = 5) type packaging.
- SEI function is disabled by default on the Web. Must call SetSEIConfig to set the SEI type before publishing and playing stream to enable the SEI function.
- Only when developers use third-party decoders to decode SEI do they need to change the SEI type to UserUnregister.
ZegoSEIConfig seiConfig = new ZegoSEIConfig();
// Use H.264's SEI (nalu type = 6, payload type = 5) type packaging. Because the video encoder itself will generate SEI with payload type 5, or when using video files to publish stream, such SEI may also exist in video files. Therefore, when using this type, users need to pass uuid + content as a buffer to the SEI sending interface; at this time, to distinguish from the SEI generated by the video encoder itself, the App can fill in a business-specific uuid (uuid length is 16 bytes) when sending this type of SEI. When the receiving party uses the SDK to parse SEI with payload type 5, it will filter out SEI with matching uuid based on the set filter string and throw it to the business. If no filter string is set, the SDK will throw all received SEI to the developer.
seiConfig.type = ZegoSEIType.ZegoDefined;
engine.SetSEIConfig(seiConfig);
// Set uuid filter field through advancedConfig. After setting, the SDK will only throw SEI whose first 12 bytes are the uuid set by the developer.
ZegoEngineConfig engineConfig = new ZegoEngineConfig();
engineConfig.advancedConfig = new Dictionary<string, string>();
// Other users receive SEI information through [onPlayerRecvSEI], the first 12 bytes must be zegozegozego, others will be filtered.
engineConfig.advancedConfig.Add("unregister_sei_filter", "zegozegozego");
ZegoExpressEngine.SetEngineConfig(engineConfig);
// Start publishing stream.
engine.StartPublishingStream("STREAM_ID");2 Publishing End
- Call the CreateEngine interface to create the engine object.
- Call the LoginRoom interface to log in to the room.
- Call the StartPublishingStream interface to publish stream.
- After successfully publishing stream, call the SendSEI interface to send SEI information.
// Define SDK engine object.
ZegoExpressEngine engine;
ZegoEngineProfile profile = new ZegoEngineProfile();
// Please obtain through official website registration, format is 123456789L.
profile.appID = appID;
// Please obtain through official website registration, format is: @"0123456789012345678901234567890123456789012345678901234567890123" (total 64 characters). Web should not pass in appSign, needs to use token authentication.
profile.appSign = appSign;
// General scenario access.
profile.scenario = ZegoScenario.DEFAULT;
// Create engine.
engine = ZegoExpressEngine.CreateEngine(profile);
// ...... Omitted step 1 set SEI type code here.
// Log in to room.
engine.LoginRoom("roomid", new ZegoUser("userid_1"));
// Publish stream.
engine.StartPublishingStream("streamid");
// Developer's other business logic.
...;
// Send SEI information at the timing required by the business scenario.
string str = "zegozegozego Hello, World!";
byte[] utf8Bytes = Encoding.UTF8.GetBytes(str);
engine.SendSEI(utf8Bytes);3 Playing End
- Call the CreateEngine interface to create the engine object.
- Implement the OnPlayerSyncRecvSEI method on the engine object to receive SEI information.
- Call the LoginRoom interface to log in to the room.
- Call the StartPlayingStream interface to play stream.
- After successfully playing stream, trigger the OnPlayerSyncRecvSEI callback after receiving the SEI information sent by the publishing end.
When playing stream, if the developer sets to play only audio stream by calling MutePlayStreamVideo or MuteAllPlayStreamVideo, SEI information cannot be received.
// Define SDK engine object.
ZegoExpressEngine engine;
ZegoEngineProfile profile = new ZegoEngineProfile();
// Please obtain through official website registration, format is 123456789L.
profile.appID = appID;
// Please obtain through official website registration, format is: @"0123456789012345678901234567890123456789012345678901234567890123" (total 64 characters). Web should not pass in appSign, needs to use token authentication.
profile.appSign = appSign;
// General scenario access.
profile.scenario = ZegoScenario.DEFAULT;
// Create engine.
engine = ZegoExpressEngine.CreateEngine(profile);
// ...... Omitted step 1 set SEI type code here.
// Implement the event callback function for onPlayerSyncRecvSEI.
engine.onPlayerSyncRecvSEI += (string streamID, byte[] data) => {
// Implement business scenario related logic here, such as displaying related UI, etc.
};
// Log in to room.
engine.LoginRoom(("roomid", new ZegoUser("userid_2"));
// Play stream, canvas is the object of the ZegoCanvas type index UI rendering control.
engine.StartPlayingStream("streamid");
// Developer's other business logic.
...;Related Documentation
How to understand and use SEI (Supplemental Enhancement Information)?