Convey extra information using SEI

Introduction

In audio and video streaming, supplemental enhancement information (SEI) is the text data inserted into the audio and video bitstream to convey extra information, which can be received in accurate synchronization with the related audio and video content.

You can use SEI for various purposes. For example:

• To include the video layout information in the stream.
• To send lyrics text in synchronization with the audio and video content.
• To send quiz information in live quizzing use cases.

Prerequisites

Before you begin, make sure you complete the following:

• Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.

• Refer to the Quick Start doc to complete the SDK integration and basic function implementation.

Implementation process

The following diagram shows the API call sequence of sending and receiving SEI messages.

On the stream publisher side, follow these steps to send SEI messages:

1. Call the createEngineWithProfile method to create a ZegoExpressEngine instance.

2. Call the loginRoom method to log in to a room.

3. Call the startPublishingStream method to start publishing a stream.

4. After the stream publishing starts successfully, call the sendSEI method to send SEI messages when needed.

On the stream subscriber side, follow these steps to receive SEI messages:

1. Call the createEngineWithProfile method to create a ZegoExpressEngine instance.
2. Create a ZegoEventHandler instance, and override the onPlayerRecvSEI method to implement your SEI message processing logic.
3. Call the setEventHandler method with the ZegoEventHandler instance created in the previous step passed in as the eventHandler parameter.
4. Call the loginRoom method to log in to a room.
5. Call the startPlayingStream method to start playing a stream.
6. After the stream playback starts successfully, when an SEI message is received, the SDK triggers the onPlayerRecvSEI callback to process the message.

Optional: Set the SEI type

Send SEI messages

On the stream publisher side, after the stream publishing starts successfully, you can call the sendSEI method to send SEI messages when needed.

• Funcation prototype:

  /// Sends supplemental enhancement information (SEI)
  ///
  /// You can use this method to send extra text information alongside the audio and video stream being published for various purposes. 
  /// For example, you can use SEI to include video layout information in a video stream or send lyrics text in synchronization with the audio and video content.
  /// After the stream publisher sends an SEI message, the stream subscribers can receive it by listening for the [onPlayerRecvSEI] callback.
  /// SEI messages are packaged as part of the bitstream of the audio/video frames, which can be lost because of network problems. To solve such potential issues, you can send the same SEI message several times with a limited frequency. 
  /// Frequency limit: No more than 30 times per second.
  /// The maximum length of the SEI data is 4096 bytes.
  ///
  /// @param data: SEI data
  - (void)sendSEI:(NSData *)data;

• Sample code:

  ZegoEngineProfile *profile = [ZegoEngineProfile new];
  // The AppID value you get from the ZEGO Admin console.
  profile.appID = appID; 
  // Use the general scenario.
  profile.scenario = ZegoScenarioGeneral; 
  // Create a ZegoExpressEngine instance and set eventHandler to [self]. If eventHandler is set to [nil], no callback will be received. You can set up the event handler later by calling the [-setEventHandler:] method.
  self.engine = [ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
  // Log in to a room.
  [self.engine loginRoom:roomID user:[ZegoUser userWithUserID:userID userName:userName]];
  // Start publishing a stream.
  [self.engine startPublishingStream:publishStreamID];
  // Implement other business logic.
  ...;
  // Send SEI messages when needed.
  char *str = "1234567\0";
  [self.manager sendSEI:[NSData dataWithBytes:str length:7 ]];

Receive SEI messages

On the stream subscriber side, after the stream playback starts successfully, the SDK triggers the onPlayerRecvSEI when an SEI message is received.

• Function prototype:

  /// The callback triggered when a supplemental enhancement information (SEI) message is received
  ///
  /// After the stream playback starts successfully, the SDK triggers this callback when an SEI message is received.
  /// If a subscriber subscribes to the audio stream only, it can't receive any SEI messages.
  ///
  /// @param data: SEI content.
  /// @param streamID: Stream ID.
  - (void)onPlayerRecvSEI:(NSData *)data streamID:(NSString *)streamID;

• Sample code:

  @implementation xx
   // Listen for the callback that is triggered when SEI messages are received.
  - (void)onPlayerRecvSEI:(NSData *)data streamID:(NSString *)streamID {
       // Implement your SEI processing logic. For example, present the related UI components.
       // ...;
  }
   // Override other callback methods.
      ...
  @end
  
  ZegoEngineProfile *profile = [ZegoEngineProfile new];
  // The AppID value you get from the ZEGO Admin console.
  profile.appID = appID; 
  // Use the general scenario.
  profile.scenario = ZegoScenarioGeneral; 
  // Create a ZegoExpressEngine instance and set eventHandler to [self]. If eventHandler is set to [nil], no callback will be received. You can set up the event handler later by calling the [-setEventHandler:] method.
  self.engine = [ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
  
  // Set up the event handler delegate.
  [self.engine setEventHandler:self];
  // Log in to a room.
  [self.engine loginRoom:roomID user:[ZegoUser userWithUserID:userID userName:userName]];
  // Start playing a stream.
  [self.engine startPlayingStream:self.firstStreamID canvas:firstPlayCanvas];
  // Implement other business logic.
  ...;