Media Supplemental Enhancement Information (SEI)

2026-03-05

Function Overview

In audio/video streaming applications, in addition to publishing and playing audio/video content through streaming channels, you can also use stream SEI (Supplemental Enhancement Information) to package text information with audio/video content through the streaming channel, publish from the host side (publishing side), and receive on the audience side (playing side), thereby achieving precise synchronization between text data and audio/video content.

It can generally be used in application scenarios such as precise video layout, remote lyric synchronization, live quiz, etc.

Note

For SEI-related concepts and principles, please refer to How to Understand and Use SEI (Media Supplemental Enhancement Information).

Example Source Code Download

Please refer to Download Example Source Code to obtain the source code.

For related source code, please view files in the "lib\topics\OtherFunctions\sei" directory.

Prerequisites

Before implementing SEI functionality, ensure that:

You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
You have integrated the ZEGO Express SDK into your project and implemented basic audio/video stream publishing and playing functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

The function of sending and receiving SEI information requires sending SEI information on the publishing side and receiving SEI information on the playing side, as shown below:

Publishing side:

Call the createEngineWithProfile interface to create an engine object.
Call the loginRoom interface to log into the room.
Call the startPublishingStream interface to publish stream.
After successfully publishing the stream, call the sendSEI interface to send SEI information.

Playing side:

Call the createEngineWithProfile interface to create an engine object.
Implement the ZegoExpressEngine.onPlayerRecvSEI callback.
Call the loginRoom interface to log into the room.
Call the startPlayingStream interface to play stream.
After successfully playing the stream, the onPlayerRecvSEI callback is triggered after receiving SEI information sent from the publishing side.

Note

When playing stream, if the developer sets to play only audio stream by calling mutePlayStreamVideo or muteAllPlayStreamVideo interfaces, SEI information cannot be received.

1 (Optional) Set SEI Type

Set SEI Type

Since the SDK uses ZEGO's self-defined SEI (nalu type = 6, payload type = 243) type packaging by default, and this type is not specified in the SEI standard, there is no conflict with SEI in video encoders or video files.

However, when developers need to use third-party decoders to decode (such as FFmpeg), it will result in inability to decode the correct SEI. At this time, before publishing stream, you need to call the setSEIConfig interface to change the SEI type sent by the SDK, using UserUnregister's SEI (nalu type = 6, payload type = 5) type packaging.

Note

This step is only required when developers use third-party decoders to decode SEI.

Interface prototype

/**
 * Set media enhancement supplemental information (SEI) type.
 *
 * Must be set before publishing stream.
 *
 * @param config SEI configuration properties. Default uses ZEGO-defined SEI type.
 */
Future<void> setSEIConfig(ZegoSEIConfig config);

/**
 * Set media enhancement supplemental information (SEI) type.
 *
 * Must be set before publishing stream.
 *
 * @param config SEI configuration properties. Default uses ZEGO-defined SEI type.
 */
Future<void> setSEIConfig(ZegoSEIConfig config);

Usage example

var engineConfig = ZegoEngineConfig();
// Set uuid filter field through advancedConfig. After setting, SDK will only throw SEI whose first 12 bytes are the uuid set by the developer.
// The first 12 bytes of SEI information received by other users through [onPlayerRecvSEI] must be zegozegozego, others will be filtered.
engineConfig.advancedConfig = {"unregister_sei_filter":"zegozegozego"};
await ZegoExpressEngine.setEngineConfig(engineConfig);

// appID and appSign are obtained through official website registration, format: 123456789L.
// ZegoScenario.General is for general scenario access.
var profile = ZegoEngineProfile(appID, ZegoScenario.General, appSign: appSign);
// Create engine.
await ZegoExpressEngine.createEngineWithProfile(profile);
// Log into room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));

// Use H.264's SEI (nalu type = 6, payload type = 5) type packaging. Because video encoders themselves will generate SEI with payload type 5, or when using video files to publish stream, such SEI may also exist in video files. So when using this type, users need to pass uuid + content as buffer to the SEI sending interface; at this time, to distinguish from SEI generated by the video encoder itself, the App can fill in business-specific uuid when sending this type of SEI (uuid length is 16 bytes). When the receiving side uses SDK to parse SEI with payload type 5, it will filter out SEI with matching uuid according to the set filter string and throw it to the business. If no filter string is set, SDK will throw all received SEI to the developer.
var seiConfig = ZegoSEIConfig(ZegoSEIType.UserUnregister);
await ZegoExpressEngine.instance.setSEIConfig(seiConfig);

// Start publishing stream.
ZegoExpressEngine.instance.startPublishingStream("STREAM_ID");

var engineConfig = ZegoEngineConfig();
// Set uuid filter field through advancedConfig. After setting, SDK will only throw SEI whose first 12 bytes are the uuid set by the developer.
// The first 12 bytes of SEI information received by other users through [onPlayerRecvSEI] must be zegozegozego, others will be filtered.
engineConfig.advancedConfig = {"unregister_sei_filter":"zegozegozego"};
await ZegoExpressEngine.setEngineConfig(engineConfig);

// appID and appSign are obtained through official website registration, format: 123456789L.
// ZegoScenario.General is for general scenario access.
var profile = ZegoEngineProfile(appID, ZegoScenario.General, appSign: appSign);
// Create engine.
await ZegoExpressEngine.createEngineWithProfile(profile);
// Log into room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));

// Use H.264's SEI (nalu type = 6, payload type = 5) type packaging. Because video encoders themselves will generate SEI with payload type 5, or when using video files to publish stream, such SEI may also exist in video files. So when using this type, users need to pass uuid + content as buffer to the SEI sending interface; at this time, to distinguish from SEI generated by the video encoder itself, the App can fill in business-specific uuid when sending this type of SEI (uuid length is 16 bytes). When the receiving side uses SDK to parse SEI with payload type 5, it will filter out SEI with matching uuid according to the set filter string and throw it to the business. If no filter string is set, SDK will throw all received SEI to the developer.
var seiConfig = ZegoSEIConfig(ZegoSEIType.UserUnregister);
await ZegoExpressEngine.instance.setSEIConfig(seiConfig);

// Start publishing stream.
ZegoExpressEngine.instance.startPublishingStream("STREAM_ID");

2 Publishing Side

The interface for sending SEI information needs to be called after successfully publishing the stream. The interface prototype is as follows:

Interface prototype

/**
 * Send media enhancement supplemental information.
 *
 * This interface can send streaming enhancement supplemental information to synchronize some other additional information while the developer is publishing audio/video stream data.
 * Generally, in scenarios such as synchronizing music lyrics or precise video layout, you can choose to use sending SEI.
 * After the publishing side sends SEI, the playing side can obtain SEI content by listening to the [onPlayerRecvSEI] callback.
 * Since SEI information follows video frames or audio frames, frame loss may occur due to network issues, so SEI information may also be lost. To solve this situation, send multiple times within the frequency limit.
 * Frequency limit: Do not exceed 30 times per second.
 * SEI data length limit is 4096 bytes.
 * @param data SEI content.
 * @param dataLength SEI content length.
 * @param channel  Publishing channel.
 */
Future<void> sendSEI(Uint8List data, int dataLength, {ZegoPublishChannel? channel});

/**
 * Send media enhancement supplemental information.
 *
 * This interface can send streaming enhancement supplemental information to synchronize some other additional information while the developer is publishing audio/video stream data.
 * Generally, in scenarios such as synchronizing music lyrics or precise video layout, you can choose to use sending SEI.
 * After the publishing side sends SEI, the playing side can obtain SEI content by listening to the [onPlayerRecvSEI] callback.
 * Since SEI information follows video frames or audio frames, frame loss may occur due to network issues, so SEI information may also be lost. To solve this situation, send multiple times within the frequency limit.
 * Frequency limit: Do not exceed 30 times per second.
 * SEI data length limit is 4096 bytes.
 * @param data SEI content.
 * @param dataLength SEI content length.
 * @param channel  Publishing channel.
 */
Future<void> sendSEI(Uint8List data, int dataLength, {ZegoPublishChannel? channel});

Usage example

// appID and appSign are obtained through official website registration, format: 123456789L.
// ZegoScenario.General is for general scenario access.
var profile = ZegoEngineProfile(appID, ZegoScenario.General, appSign: appSign));
// Create engine.
await ZegoExpressEngine.createEngineWithProfile(profile);
// When real-time audio/video SDK is used for pure audio scenarios, you can disable the camera, so camera permissions and video stream publishing will not be required.
// ZegoExpressEngine.instance.enableCamera(false);
// Log into room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));
// Publish stream.
await ZegoExpressEngine.instance.startPublishingStream("streamid");
// Developer's other business logic.
...;
// Send SEI information at the timing required by the business scenario.
var seiData = Uint8List.fromList(utf8.encode('12345'));
ZegoExpressEngine.instance.sendSEI(seiData, seiData.length);

// appID and appSign are obtained through official website registration, format: 123456789L.
// ZegoScenario.General is for general scenario access.
var profile = ZegoEngineProfile(appID, ZegoScenario.General, appSign: appSign));
// Create engine.
await ZegoExpressEngine.createEngineWithProfile(profile);
// When real-time audio/video SDK is used for pure audio scenarios, you can disable the camera, so camera permissions and video stream publishing will not be required.
// ZegoExpressEngine.instance.enableCamera(false);
// Log into room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_1", "username_1"));
// Publish stream.
await ZegoExpressEngine.instance.startPublishingStream("streamid");
// Developer's other business logic.
...;
// Send SEI information at the timing required by the business scenario.
var seiData = Uint8List.fromList(utf8.encode('12345'));
ZegoExpressEngine.instance.sendSEI(seiData, seiData.length);

3 Playing Side

The callback interface for receiving SEI information needs to be triggered after successfully playing the stream. The interface prototype is as follows:

Interface prototype

/**
 * Received SEI content from remote stream.
 *
 * After successfully playing stream, when the remote stream calls sendSEI, this callback will be received.
 * If only playing pure audio stream, SEI information sent from the publishing side will not be received.
 * @param streamID Stream ID of the playing stream.
 * @param data SEI content.
 */
static void Function(String streamID, Uint8List data)? onPlayerRecvSEI;

/**
 * Received SEI content from remote stream.
 *
 * After successfully playing stream, when the remote stream calls sendSEI, this callback will be received.
 * If only playing pure audio stream, SEI information sent from the publishing side will not be received.
 * @param streamID Stream ID of the playing stream.
 * @param data SEI content.
 */
static void Function(String streamID, Uint8List data)? onPlayerRecvSEI;

Usage example

// appID and appSign are obtained through official website registration, format: 123456789L.
// ZegoScenario.General is for general scenario access.
var profile = ZegoEngineProfile(appID, ZegoScenario.General, appSign: appSign));
// Create engine.
await ZegoExpressEngine.createEngineWithProfile(profile);

// When real-time audio/video SDK is used for pure audio scenarios, you can disable the camera, so camera permissions and video stream publishing will not be required.
// ZegoExpressEngine.instance.enableCamera(false);

// Listen to the callback for receiving SEI information. This callback will be triggered when the sending side calls sendSEI to send information.
ZegoExpressEngine.onPlayerRecvSEI = (String streamID, Uint8List data) {
   // Implement business scenario-related logic here, such as displaying relevant UI, etc.
};
// Log into room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_2", "username_2"));
// Play stream. canvas is the object of ZegoCanvas type index UI rendering control. When only using real-time audio/video SDK or real-time audio/video scenarios, the canvas parameter must be passed.
await ZegoExpressEngine.instance.startPlayingStream("streamid", canvas: canvas);
// If using real-time audio SDK or pure audio scenarios, the canvas parameter is not required.
// ZegoExpressEngine.instance.startPlayingStream("streamid");
// Developer's other business logic.
...;

// appID and appSign are obtained through official website registration, format: 123456789L.
// ZegoScenario.General is for general scenario access.
var profile = ZegoEngineProfile(appID, ZegoScenario.General, appSign: appSign));
// Create engine.
await ZegoExpressEngine.createEngineWithProfile(profile);

// When real-time audio/video SDK is used for pure audio scenarios, you can disable the camera, so camera permissions and video stream publishing will not be required.
// ZegoExpressEngine.instance.enableCamera(false);

// Listen to the callback for receiving SEI information. This callback will be triggered when the sending side calls sendSEI to send information.
ZegoExpressEngine.onPlayerRecvSEI = (String streamID, Uint8List data) {
   // Implement business scenario-related logic here, such as displaying relevant UI, etc.
};
// Log into room.
await ZegoExpressEngine.instance.loginRoom("roomid", ZegoUser("userid_2", "username_2"));
// Play stream. canvas is the object of ZegoCanvas type index UI rendering control. When only using real-time audio/video SDK or real-time audio/video scenarios, the canvas parameter must be passed.
await ZegoExpressEngine.instance.startPlayingStream("streamid", canvas: canvas);
// If using real-time audio SDK or pure audio scenarios, the canvas parameter is not required.
// ZegoExpressEngine.instance.startPlayingStream("streamid");
// Developer's other business logic.
...;

How to Understand and Use SEI (Media Supplemental Enhancement Information)?

Media Supplemental Enhancement Information (SEI)

Function Overview

Example Source Code Download

Prerequisites

Usage Steps

1 (Optional) Set SEI Type

2 Publishing Side

3 Playing Side

Related Documents