Supplemental Enhancement Information (SEI)

Introduction

In audio/video streaming applications, in addition to publishing and playing audio/video content through streaming channels, you can also use SEI (Supplemental Enhancement Information) to package text information together with audio/video content through the streaming channel, sent from the host side (publishing side) and received by the audience side (playing side), achieving precise synchronization between text data and audio/video content.

It is commonly used for application scenarios such as remote lyric synchronization and live quizzes.

Note

For SEI-related concepts and principles, please refer to How to Understand and Use SEI (Supplemental Enhancement Information).

Prerequisites

Before implementing SEI functionality, please ensure:

The ZEGO Express SDK has been integrated in your project and basic real-time voice functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementing Flow.
A project has been created in the ZEGOCLOUD Console and a valid AppID has been obtained.

Usage Steps

The SEI information sending and receiving functionality requires the publishing and playing sides to work together to demonstrate the effect, meaning SEI information needs to be sent from the publishing side and received by the playing side. The following content will describe how to use the SEI sending and receiving functionality on each side.

The host publishing flow for sending SEI messages is as follows:

Call the loginRoom interface to log in to the room.
Call the startPublishingStream interface to publish the stream.
After successful publishing, call the sendSEI interface to send SEI information.

The audience playing flow for receiving SEI messages is as follows:

Call on to listen for the playerRecvSEI callback.
Call the loginRoom interface to log in to the room.
Call the startPlayingStream interface to play the stream.
After successful playing, when SEI information sent from the publishing side is received, the playerRecvSEI callback will be triggered.

Warning

SEI functionality currently only supports Chrome 86 and above browsers.
When playing streams through CDN, you need to use a third-party player such as flv.js, but third-party players do not support parsing SEI by default.

(Optional) Set SEI Additional Information

Set SEI Additional Information

Since the SDK uses ZEGO's self-defined SEI (nalu type = 6, payload type = 243) type for packaging by default, and this type is not specified in the SEI standard, it does not conflict with SEI in video encoders or video files.
However, when developers need to use third-party decoders for decoding (such as FFmpeg), it will result in the inability to extract the correct SEI. In this case, when publishing the stream startPublishingStream, you need to specify the sent SEI type as UserUnregister SEI (nalu type = 6, payload type = 5), and call the setSEIConfig interface before publishing and before playing to set the uuid (UserUnregisterID) to distinguish whether the SEI is generated by the video encoder itself or is business SEI.
When the App sends this type of SEI, it can fill in a business-specific uuid (length is 16 bytes). When the receiver uses the SDK to parse SEI with payload type 5, it will filter out SEI with matching uuid based on the set filter string and throw it to the business. If no filter string is set, the SDK will throw all received SEI to the developer.

Note

This step only needs to be performed when developers use third-party decoders to decode SEI.

let appID = ;
let server = "";

// Initialize
const zg = new ZegoExpressEngine(appID, server);

zg.setSEIConfig({
    // Custom specific string to filter out business SEI
    unregisterSEIFilter: "zegozegozegozego"
});

let appID = ;
let server = "";

// Initialize
const zg = new ZegoExpressEngine(appID, server);

zg.setSEIConfig({
    // Custom specific string to filter out business SEI
    unregisterSEIFilter: "zegozegozegozego"
});

Publishing Side

The interface for sending SEI information needs to be called after successful stream publishing.

let appID = ;
let server = "";

// Initialize
const zg = new ZegoExpressEngine(appID, server);

// User ID, custom
let userID = "user_" + new Date().getTime();
// roomID, custom
let roomID = "0001";
// Authentication token
let token = "";
// Publishing stream ID
let publishStreamID = "00001";

// Log in to room
zg.loginRoom(roomID, token, { userID, userName: userID }, { userUpdate: true }).then(result => {
     if (result == true) {
        console.log("login success")
     }
});

// Create local stream preview
const localStream = await zg.createZegoStream({camera:{audio:true, video:false}});

zg.on("publisherStateUpdate", async result => {
    if (result.state === "PUBLISHING") {
        // SEI content to be sent, example as follows:
        // Timestamp
        const ts = Math.ceil(new Date().getTime() / 1000);
        // Convert to bytes
        const u = new Uint8Array(4);
        u[0] = (ts >> 24) & 0xff;
        u[1] = (ts >> 16) & 0xff;
        u[2] = (ts >> 8) & 0xff;
        u[3] = ts & 0xff;

        // After successful publishing, send SEI
        zg.sendSEI(streamID, u);
    }

});

// Publish stream
zg.startPublishingStream(publishStreamID, localStream, {
  roomID,
  isSEIStart: true, // Enable sending SEI
  SEIType: 0 // Default is 0, representing payload type = 243
});

let appID = ;
let server = "";

// Initialize
const zg = new ZegoExpressEngine(appID, server);

// User ID, custom
let userID = "user_" + new Date().getTime();
// roomID, custom
let roomID = "0001";
// Authentication token
let token = "";
// Publishing stream ID
let publishStreamID = "00001";

// Log in to room
zg.loginRoom(roomID, token, { userID, userName: userID }, { userUpdate: true }).then(result => {
     if (result == true) {
        console.log("login success")
     }
});

// Create local stream preview
const localStream = await zg.createZegoStream({camera:{audio:true, video:false}});

zg.on("publisherStateUpdate", async result => {
    if (result.state === "PUBLISHING") {
        // SEI content to be sent, example as follows:
        // Timestamp
        const ts = Math.ceil(new Date().getTime() / 1000);
        // Convert to bytes
        const u = new Uint8Array(4);
        u[0] = (ts >> 24) & 0xff;
        u[1] = (ts >> 16) & 0xff;
        u[2] = (ts >> 8) & 0xff;
        u[3] = ts & 0xff;

        // After successful publishing, send SEI
        zg.sendSEI(streamID, u);
    }

});

// Publish stream
zg.startPublishingStream(publishStreamID, localStream, {
  roomID,
  isSEIStart: true, // Enable sending SEI
  SEIType: 0 // Default is 0, representing payload type = 243
});

Playing Side

The callback interface for receiving SEI information will be triggered after successful stream playing.

let appID = ;
let server = "";

// Initialize
const zg = new ZegoExpressEngine(appID, server);

// User ID, custom
let userID = "user_" + new Date().getTime();
// roomID, custom
let roomID = "0001";
// Authentication token
let token = "";
// Playing stream ID, generally obtained through roomStreamUpdate callback
let playStreamID = "00001";

// Listen for SEI callback
zg.on("playerRecvSEI", (streamID, uintArray) => {
    let offset = 0;
    // The first 4 bytes of the received SEI content represent the sent SEI type, convert it to number,
    // 1004 represents payload type = 5, 1005 represents payload type = 243
    let mediaSideInfoType = 0;
    mediaSideInfoType = uintArray[offset++] << 24;
    mediaSideInfoType |= uintArray[offset++] << 16;
    mediaSideInfoType |= uintArray[offset++] << 8;
    mediaSideInfoType |= uintArray[offset++];

    // Based on the sent SEI, parse the SEI content, example as follows:
    const view = new DataView(uintArray.buffer);
    let i = 4;
    let ts = 0;
    ts = view.getUint8(i++) << 24;
    ts |= view.getUint8(i++) << 16;
    ts |= view.getUint8(i++) << 8;
    ts |= view.getUint8(i++);

    console.log("recv " + streamID + " " + mediaSideInfoType + " " + ts);
}

// Log in to room
zg.loginRoom(roomID, token, { userID, userName: userID }, { userUpdate: true }).then(result => {
     if (result == true) {
        console.log("login success")
     }
});

// Play stream
zg.startPlayingStream(playStreamID, {
    isSEIStart: true // Enable parsing SEI
}).then(stream => {

}).catch(err => {

});

let appID = ;
let server = "";

// Initialize
const zg = new ZegoExpressEngine(appID, server);

// User ID, custom
let userID = "user_" + new Date().getTime();
// roomID, custom
let roomID = "0001";
// Authentication token
let token = "";
// Playing stream ID, generally obtained through roomStreamUpdate callback
let playStreamID = "00001";

// Listen for SEI callback
zg.on("playerRecvSEI", (streamID, uintArray) => {
    let offset = 0;
    // The first 4 bytes of the received SEI content represent the sent SEI type, convert it to number,
    // 1004 represents payload type = 5, 1005 represents payload type = 243
    let mediaSideInfoType = 0;
    mediaSideInfoType = uintArray[offset++] << 24;
    mediaSideInfoType |= uintArray[offset++] << 16;
    mediaSideInfoType |= uintArray[offset++] << 8;
    mediaSideInfoType |= uintArray[offset++];

    // Based on the sent SEI, parse the SEI content, example as follows:
    const view = new DataView(uintArray.buffer);
    let i = 4;
    let ts = 0;
    ts = view.getUint8(i++) << 24;
    ts |= view.getUint8(i++) << 16;
    ts |= view.getUint8(i++) << 8;
    ts |= view.getUint8(i++);

    console.log("recv " + streamID + " " + mediaSideInfoType + " " + ts);
}

// Log in to room
zg.loginRoom(roomID, token, { userID, userName: userID }, { userUpdate: true }).then(result => {
     if (result == true) {
        console.log("login success")
     }
});

// Play stream
zg.startPlayingStream(playStreamID, {
    isSEIStart: true // Enable parsing SEI
}).then(stream => {

}).catch(err => {

});

How to Understand and Use SEI (Supplemental Enhancement Information)?