Supplemental Enhancement Information (SEI)

Feature Overview

In audio and video streaming applications, in addition to publishing and playing audio and video content through streaming channels, you can also use stream SEI (Supplemental Enhancement Information) to package text information together with audio and video content through the streaming channel, publish it from the broadcaster end (publish stream end), and receive it at the audience end (play stream end), thereby achieving precise synchronization between text data and audio and video content.

Generally can be used for application scenarios such as precise layout of video screens, remote lyric synchronization, and live quiz.

Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Others/src/main/java/im/zego/others/sei" directory.

Prerequisites

Before implementing SEI functionality, please ensure:

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
• You have integrated ZEGO Express SDK in your project and implemented basic audio and video publish and play stream functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

The function of sending and receiving SEI information requires sending SEI information at the publish stream end and receiving SEI information at the play stream end, as shown in the following figure:

Publish stream end:

1. Call the createEngine interface to create an engine object.

2. Call the loginRoom interface to log in to the room.

3. Call the startPublishingStream interface to publish stream.

4. After successful publishing, call the sendSEI interface to send SEI information.

Play stream end:

1. Call the createEngine interface to create an engine object.

2. Create an IZegoEventHandler object and override the onPlayerRecvSEI method for receiving SEI information. Call the setEventHandler interface and pass in the created IZegoEventHandler to listen to the onPlayerRecvSEI callback.

3. Call the loginRoom interface to log in to the room.

4. Call the startPlayingStream interface to play stream.

5. After successfully playing stream, when receiving SEI information sent by the publish stream end, the onPlayerRecvSEI callback will be triggered.

   Note

   When playing stream, if the developer has set to play only audio stream by calling mutePlayStreamVideo or muteAllPlayStreamVideo interfaces, SEI information cannot be received.

1 (Optional) Set SEI type

ZegoSEIConfig seiConfig = new ZegoSEIConfig();
// Use H.264's SEI (nalu type = 6, payload type = 5) type for packaging. Since the video encoder itself will generate SEI with payload type of 5, or when using video files to publish stream, such SEI may also exist in the video file. Therefore, when using this type, users need to put uuid + content as a buffer into the SEI sending interface. At this time, to distinguish from the SEI generated by the video encoder itself, the App can fill in a business-specific uuid (uuid length is 16 bytes) when sending this type of SEI. When the receiver uses the SDK to parse SEI with payload type of 5, it will filter out SEI with matching uuid according to the set filter string and throw it to the business. If no filter string is set, the SDK will throw all received SEI to the developer.
seiConfig.type = ZegoSEIType.USER_UNREGISTER;

engine.setSEIConfig(seiConfig);

// Set uuid filter field through advancedConfig. After setting, the SDK will only throw SEI whose first 12 bytes are the uuid set by the developer.
ZegoEngineConfig engineConfig = new ZegoEngineConfig();
// The first 12 bytes of SEI information received by other users through [onPlayerRecvSEI] must be zegozegozego, and others will be filtered.
engineConfig.advancedConfig.put("unregister_sei_filter", "zegozegozego");
ZegoExpressEngine.setEngineConfig(engineConfig);

// Start publishing stream.
engine.startPublishingStream("STREAM_ID");

ZegoSEIConfig seiConfig = new ZegoSEIConfig();
// Use H.264's SEI (nalu type = 6, payload type = 5) type for packaging. Since the video encoder itself will generate SEI with payload type of 5, or when using video files to publish stream, such SEI may also exist in the video file. Therefore, when using this type, users need to put uuid + content as a buffer into the SEI sending interface. At this time, to distinguish from the SEI generated by the video encoder itself, the App can fill in a business-specific uuid (uuid length is 16 bytes) when sending this type of SEI. When the receiver uses the SDK to parse SEI with payload type of 5, it will filter out SEI with matching uuid according to the set filter string and throw it to the business. If no filter string is set, the SDK will throw all received SEI to the developer.
seiConfig.type = ZegoSEIType.USER_UNREGISTER;

engine.setSEIConfig(seiConfig);

// Set uuid filter field through advancedConfig. After setting, the SDK will only throw SEI whose first 12 bytes are the uuid set by the developer.
ZegoEngineConfig engineConfig = new ZegoEngineConfig();
// The first 12 bytes of SEI information received by other users through [onPlayerRecvSEI] must be zegozegozego, and others will be filtered.
engineConfig.advancedConfig.put("unregister_sei_filter", "zegozegozego");
ZegoExpressEngine.setEngineConfig(engineConfig);

// Start publishing stream.
engine.startPublishingStream("STREAM_ID");

2 Publish stream end

The interface for sending SEI information needs to be called after successful publishing. The interface prototype is as follows:

• Interface prototype

  /**
   * Send media enhancement supplemental information.
   *
   * This interface allows developers to send streaming media enhancement supplemental information to synchronize some other additional information while publishing audio and video stream data.
   * Generally, scenarios such as synchronizing music lyrics or precise layout of video screens can choose to use sending SEI.
   * After the publish stream end sends SEI, the play stream end can obtain SEI content by listening to the [onPlayerRecvSEI] callback.
   * Since SEI information follows video frames or audio frames, it may be lost due to network issues causing frame loss. Therefore, SEI information may also be lost. To solve this situation, it should be sent multiple times within the frequency limit.
   * Frequency limit: Do not exceed 30 times per second.
   * SEI data length limit is 4096 bytes.
   * @param data SEI content.
   */
  public void sendSEI(byte[] data)

  /**
   * Send media enhancement supplemental information.
   *
   * This interface allows developers to send streaming media enhancement supplemental information to synchronize some other additional information while publishing audio and video stream data.
   * Generally, scenarios such as synchronizing music lyrics or precise layout of video screens can choose to use sending SEI.
   * After the publish stream end sends SEI, the play stream end can obtain SEI content by listening to the [onPlayerRecvSEI] callback.
   * Since SEI information follows video frames or audio frames, it may be lost due to network issues causing frame loss. Therefore, SEI information may also be lost. To solve this situation, it should be sent multiple times within the frequency limit.
   * Frequency limit: Do not exceed 30 times per second.
   * SEI data length limit is 4096 bytes.
   * @param data SEI content.
   */
  public void sendSEI(byte[] data)

• Usage example

  // Define SDK engine object.
  ZegoExpressEngine engine;
  
  ZegoEngineProfile profile = new ZegoEngineProfile();
  // Please obtain through official website registration, format is 123456789L.
  profile.appID = appID;
  // Please obtain through official website registration, format is: @"0123456789012345678901234567890123456789012345678901234567890123" (total 64 characters).
  profile.appSign = appSign;
  // General scenario access.
  profile.scenario = ZegoScenario.DEFAULT;
  // Set app's application object.
  profile.application = getApplication();
  // Create engine.
  engine = ZegoExpressEngine.createEngine(profile, null);
  // Log in to room.
  engine.loginRoom("roomid", new ZegoUser("userid_1"));
  // Publish stream.
  engine.startPublishingStream("streamid");
  // Developer's other business logic.
  ...;
  // Send SEI information at the timing required by the business scenario.
  engine.sendSEI("12345".getBytes());

  // Define SDK engine object.
  ZegoExpressEngine engine;
  
  ZegoEngineProfile profile = new ZegoEngineProfile();
  // Please obtain through official website registration, format is 123456789L.
  profile.appID = appID;
  // Please obtain through official website registration, format is: @"0123456789012345678901234567890123456789012345678901234567890123" (total 64 characters).
  profile.appSign = appSign;
  // General scenario access.
  profile.scenario = ZegoScenario.DEFAULT;
  // Set app's application object.
  profile.application = getApplication();
  // Create engine.
  engine = ZegoExpressEngine.createEngine(profile, null);
  // Log in to room.
  engine.loginRoom("roomid", new ZegoUser("userid_1"));
  // Publish stream.
  engine.startPublishingStream("streamid");
  // Developer's other business logic.
  ...;
  // Send SEI information at the timing required by the business scenario.
  engine.sendSEI("12345".getBytes());

3 Play stream end

The callback interface for receiving SEI information needs to be triggered after successful playing. The interface prototype is as follows:

• Interface prototype

  /**
   * Received SEI content from remote stream.
   *
   * After successfully playing stream, when the remote stream calls sendSEI, this callback will be received.
   * If only playing pure audio stream, SEI information sent by the publish stream end will not be received.
   * @param streamID Stream ID of the played stream.
   * @param data SEI content.
   */
  public void onPlayerRecvSEI(String streamID, byte[] data){
  
  }

  /**
   * Received SEI content from remote stream.
   *
   * After successfully playing stream, when the remote stream calls sendSEI, this callback will be received.
   * If only playing pure audio stream, SEI information sent by the publish stream end will not be received.
   * @param streamID Stream ID of the played stream.
   * @param data SEI content.
   */
  public void onPlayerRecvSEI(String streamID, byte[] data){
  
  }

• Usage example

  // Define SDK engine object.
  ZegoExpressEngine engine;
  
  ZegoEngineProfile profile = new ZegoEngineProfile();
  // Please obtain through official website registration, format is 123456789L.
  profile.appID = appID;
  // Please obtain through official website registration, format is: @"0123456789012345678901234567890123456789012345678901234567890123" (total 64 characters).
  profile.appSign = appSign;
  // General scenario access.
  profile.scenario = ZegoScenario.DEFAULT;
  // Set app's application object.
  profile.application = getApplication();
  // Create engine.
  engine = ZegoExpressEngine.createEngine(profile, null);
  
  // Create IZegoEventHandler object and override onPlayerRecvSEI method.
  IZegoEventHandler handler = new IZegoEventHandler(){
  // Listen to other callbacks.
      ...;
  
      // Listen to callback for receiving SEI information. When the sender calls sendSEI to send information, this callback will be triggered.
      public void onPlayerRecvSEI(String streamID, byte[] data) {
      // Implement business scenario related logic here, such as displaying related UI, etc.
          ...;
      }
  
  }
  // Add the callback object to listen.
  engine.setEventHandler(handler);
  // Log in to room.
  engine.loginRoom("roomid", new ZegoUser("userid_2"));
  // Play stream, canvas is the object of the indexed UI rendering control of ZegoCanvas type.
  engine.startPlayingStream("streamid", canvas);
  // Developer's other business logic.
  ...;

  // Define SDK engine object.
  ZegoExpressEngine engine;
  
  ZegoEngineProfile profile = new ZegoEngineProfile();
  // Please obtain through official website registration, format is 123456789L.
  profile.appID = appID;
  // Please obtain through official website registration, format is: @"0123456789012345678901234567890123456789012345678901234567890123" (total 64 characters).
  profile.appSign = appSign;
  // General scenario access.
  profile.scenario = ZegoScenario.DEFAULT;
  // Set app's application object.
  profile.application = getApplication();
  // Create engine.
  engine = ZegoExpressEngine.createEngine(profile, null);
  
  // Create IZegoEventHandler object and override onPlayerRecvSEI method.
  IZegoEventHandler handler = new IZegoEventHandler(){
  // Listen to other callbacks.
      ...;
  
      // Listen to callback for receiving SEI information. When the sender calls sendSEI to send information, this callback will be triggered.
      public void onPlayerRecvSEI(String streamID, byte[] data) {
      // Implement business scenario related logic here, such as displaying related UI, etc.
          ...;
      }
  
  }
  // Add the callback object to listen.
  engine.setEventHandler(handler);
  // Log in to room.
  engine.loginRoom("roomid", new ZegoUser("userid_2"));
  // Play stream, canvas is the object of the indexed UI rendering control of ZegoCanvas type.
  engine.startPlayingStream("streamid", canvas);
  // Developer's other business logic.
  ...;

Related Documentation

How to understand and use SEI (Supplemental Enhancement Information)?