Documentation
Live Streaming
Upgrade the livestream
Advanced features
Message signaling
Convey extra information using SEI

Convey extra information using SEI

Last updated：2023-09-07 14:18

1 Introduction

In audio and video streaming, supplemental enhancement information (SEI) is the text data inserted into the audio and video bitstream to convey extra information, which can be received in accurate synchronization with the related audio and video content.

You can use SEI for various purposes. For example:

To include the video layout information in the stream.
To send lyrics text in synchronization with the audio and video content.
To send quiz information in live quizzing use cases.

For more details about SEI, see How to understand and use supplementation enhancement information.

2 Prerequisites

Before you begin, make sure you complete the following:

Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.
Refer to the Quick Start doc to complete the SDK integration and basic function implementation.

3 Implementation steps

The following diagram shows the API call sequence of sending and receiving SEI messages.

/Pics/Common/ZegoExpressEngine/send_and_recv_sei_Flutter.png

On the stream publisher side, follow these steps to send SEI messages:

Call the createEngine method to create a ZegoExpressEngine instance.
Call the loginRoom method to log in to a room.
Call the startPublishingStream method to start publishing a stream.
After the stream publishing starts successfully, call the sendSEI method to send SEI messages when needed.

On the stream subscriber side, follow these steps to receive SEI messages:

Call the createEngine method to create a ZegoExpressEngine instance.
Implement the ZegoExpressEngine.onPlayerRecvSEI callback to process SEI messages.
Call the loginRoom method to log in to a room.
Call the startPlayingStream method to start playing a stream.
After the stream playback starts successfully, when an SEI message is received, the SDK triggers the onPlayerRecvSEI callback to process the message.

3.1 Optional: Set the SEI type

Set SEI type

By default, the ZEGO Express SDK uses a ZEGO-defined SEI type (NALU Type = 6, Payload Type = 243) for packaging SEI messages. This SEI type isn't defined in the SEI types of the SDK's video codes and doesn't mix up with the existing SEI data of the source video file.

But if you use a third-party decoder like FFmpeg, the SEI messages with the ZEGO-defined SEI type can't be decoded correctly. To solve this issue, you need to call the setSEIConfig method to change the SEI type to the UserUnregister type (NALU Type = 6, Payload Type = 5).

As SEI data with Payload Type 5 can also be generated by the video encoder or may exist in the source video file, when you send SEI with Payload Type 5, you need a mechanism to differentiate your SEI data from the SEI data generated by the video encoder itself. For this purpose, you can configure a UUID (16 bytes in length) as an SEI filtering keyword for the ZegoExpressEngine. On the stream publisher side, pass "UUID + message content" as the buffer to the sendSEI method when sending SEI messages. On the stream subscriber side, the SDK only throws out those SEI messages that start with the configured UUID. If no UUID is configured, the SDK throws out all received SEI messages.

This step is required only when you use a third-party decoder to decode SEI.

Function prototype:

/**
 * Sets the supplemental enhancement information (SEI) type
 *
 * The SEI type must be set before stream publishing starts.
 *
 * @param config: The SEI configuration object. By default, the ZEGO Express SDK uses the ZEGO-defined SEI type to package SEI messages.
 */
Future<void> setSEIConfig(ZegoSEIConfig config);

Sample code:

var engineConfig = ZegoEngineConfig();
// Use `advancedConfig` to set the uuid filtering field. After setting this, the SDK will only throw SEIs whose first 12 bytes match the uuid set by the developer to the developer.
// For other users, the first 12 bytes of the SEI information they receive through [onPlayerRecvSEI] will always be "zegozegozego," and other information will be filtered out.
engineConfig.advancedConfig = {"unregister_sei_filter":"zegozegozego"};
await ZegoExpressEngine.setEngineConfig(engineConfig);

var profile = ZegoEngineProfile(appID, ZegoScenario.Default, appSign: appSign);
/** create engine */
await ZegoExpressEngine.createEngineWithProfile(profile);
// loginRoom
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));

// To package SEI using H.264 (nalu type = 6, payload type = 5), because the video encoder itself may generate a SEI with payload type 5, or when pushing video files, there may be a SEI with this payload type in the video file. In this case, the user needs to pass the uuid + content as a buffer to the SEI sending interface. To distinguish the SEI generated by the video encoder itself, when sending this type of SEI, the App can fill in a business-specific uuid (the length of the uuid is 16 bytes). When the receiver uses the SDK to parse a SEI with payload type 5, it will filter out the SEI that matches the uuid based on the filtering string set by the user and throw it to the business. If no filtering string is set, the SDK will throw all received SEIs to the developer.
var seiConfig = ZegoSEIConfig(ZegoSEIType.UserUnregister);
await ZegoExpressEngine.instance.setSEIConfig(seiConfig);

// start publishing
ZegoExpressEngine.instance.startPublishingStream("STREAM_ID");

3.2 Send SEI messages

On the stream publisher side, after the stream publishing starts successfully, you can call the sendSEI method to send SEI messages when needed.

Funcation prototype:

/**
 * Sends supplemental enhancement information (SEI)
 *
 * You can use this method to send extra text information alongside the audio and video stream being published for various purposes. 
 * For example, you can use SEI to include video layout information in a video stream or send lyrics text in synchronization with the audio and video content.
 * After the stream publisher sends an SEI message, the stream subscribers can receive it by listening for the [onPlayerRecvSEI] callback.
 * SEI messages are packaged as part of the bitstream of the audio/video frames, which can be lost because of network problems. To solve such potential issues, you can send the same SEI message several times with a limited frequency. 
 * Frequency limit: No more than 30 times per second.
 * The maximum length of the SEI data is 4096 bytes.
 * @param data: SEI data
 */
Future<void> sendSEI(Uint8List data, int dataLength, {ZegoPublishChannel? channel});

Sample code:

// Create a ZegoExpressEngine instance with your AppID and AppSign (assigned by ZEGO when you create your project in the ZEGO Admin Console), initialize it to use the testing environment and the settings for the general scenario. 
// If you don't want to set up the event handler for the moment, you can set the eventHandler parameter to null. Then, you can set it up later by calling the setEventHandler method.
var profile = ZegoEngineProfile(appID, ZegoScenario.Default, appSign: appSign);
await ZegoExpressEngine.createEngineWithProfile(profile);
// Log in to a room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));
// Start publishing a stream.
await ZegoExpressEngine.instance.startPublishingStream("streamid");
// Implement other business logic.
...;
// Send SEI messages when needed.
var seiData = Uint8List.fromList(utf8.encode('12345'));
ZegoExpressEngine.instance.sendSEI(seiData, seiData.length);

3.3 Receive SEI messages

On the stream subscriber side, after the stream playback starts successfully, the SDK triggers the onPlayerRecvSEI callback when an SEI message is received.

Function prototype:

/**
 * The callback triggered when a supplemental enhancement information (SEI) message is received
 *
 * After the stream playback starts successfully, the SDK triggers this callback when an SEI message is received.
 * If a subscriber subscribes to the audio stream only, it can't receive any SEI messages.
 * @param data: SEI content
 * @param streamID: Stream ID
 */
static void Function(String streamID, Uint8List data)? onPlayerRecvSEI;

Sample code:

// Create a ZegoExpressEngine instance with your AppID and AppSign (assigned by ZEGO when you create your project in the ZEGO Admin Console), initialize it to use the testing environment and the settings for the general scenario. 
// If you don't want to set up the event handler for the moment, you can set the eventHandler parameter to null. Then, you can set it up later by calling the setEventHandler method.
var profile = ZegoEngineProfile(appID, ZegoScenario.Default, appSign: appSign);
await ZegoExpressEngine.createEngineWithProfile(profile);

// Listen to the onPlayerRecvSEI callback that receives SEI information. 
// This callback is triggered when the sender calls `sendSEI` to send information.
ZegoExpressEngine.onPlayerRecvSEI = (String streamID, Uint8List data) {
    // Implement the business-related logic here, such as displaying related UI, etc.
};

// Log in to a room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));
// Start playing a stream.
await ZegoExpressEngine.instance.startPlayingStream("streamid");
// Implement other business logic.
...;

4 API reference

Method	Description
sendSEI	Sends a SEI message.
onPlayerRecvSEI	The callback triggered when an SEI message is received.
setSEIConfig	Sets SEI type.