Custom Video Preprocessing

Feature Introduction

Video preprocessing is a process between capture and encoding. Developers capture video data themselves. After obtaining the video data captured by the SDK, they can perform video preprocessing through the SDK's built-in basic beautification and watermark features. If the SDK cannot meet developers' needs (for example, the beautification effect cannot meet expectations), they can also use the ZEGO Effects SDK to perform special processing on the video, such as beautification, adding accessories, etc. This process is custom video preprocessing.

Compared with custom video capture, the advantage of custom video preprocessing is that developers do not need to manage device input sources, they only need to operate on the raw data thrown out by the ZEGO Express SDK, and then send it back to the ZEGO Express SDK.

Sample Source Code Download

ZEGO provides sample source code for implementing beautification features through the Effects SDK. Please refer to Using Real-time Audio and Video with AI Beautification Together.

Prerequisites

Before performing custom video preprocessing, please ensure:

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
• You have integrated the ZEGO Express SDK in the project and implemented basic audio and video publishing and playing functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation Flow.

Usage Steps

The flow and interface calls for custom video preprocessing are as follows:

1. Initialize Express SDK and log in to room.
2. Call the enableCustomVideoProcessing interface to enable custom video preprocessing feature.
3. Get raw video data and perform video preprocessing.
   1. Call the setCustomVideoProcessHandler interface to register custom video preprocessing callbacks and implement onCapturedUnprocessedCVPixelBuffer and other callback methods.
   2. Start preview and publish stream.
   3. Through the onStart callback, notify that custom video preprocessing has started. If developers use third-party beautification libraries or pre-allocate memory, they can initialize tools and allocate memory at this time.
   4. Get raw video data through the onCapturedUnprocessedCVPixelBuffer callback interface. Developers can perform related processing according to business needs: can be processed through Express SDK's built-in basic beautification, watermark and other features; can also be used with Effects SDK to implement more processing effects.
   5. After video data processing is completed, in the onCapturedUnprocessedCVPixelBuffer callback, call the sendCustomVideoProcessedCVPixelBuffer interface to send the processed video frame data to the Express SDK.
4. Other clients can play the processed video stream.
5. Stop preview and publishing stream.
6. Through the onStop callback, notify that custom video preprocessing has ended. If developers use third-party beautification libraries or pre-allocate memory, they can deinitialize tools and release memory at this time.
7. Log out of room, destroy engine, and release resources.

1 Enable custom video preprocessing

Create a custom video preprocessing ZegoCustomVideoProcessConfig object and set the bufferType property to provide video frame data types.

Currently, the iOS SDK supports the following 2 types of bufferType data types. Setting other enumeration values will not work properly:

Before starting preview and starting publishing stream, call the enableCustomVideoProcessing interface to enable the custom video preprocessing feature.

The following takes "ZegoVideoBufferTypeCVPixelBuffer" as an example to demonstrate the usage of custom video preprocessing.

ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select CVPixelBuffer type video frame data
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;

[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];

ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select CVPixelBuffer type video frame data
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;

[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];

2 Get raw video data and perform video preprocessing

Register custom video preprocessing callback

1. Use "ViewController" as the custom video preprocessing callback object and conform to the ZegoCustomVideoProcessHandler protocol.

   @interface ViewController () <ZegoEventHandler, ZegoCustomVideoProcessHandler>
   
       ......
   
   @end

   @interface ViewController () <ZegoEventHandler, ZegoCustomVideoProcessHandler>
   
       ......
   
   @end

2. Call the setCustomVideoProcessHandler interface to set the custom video capture callback.

   // Set self as the custom video preprocessing callback object
   [[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];

   // Set self as the custom video preprocessing callback object
   [[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];

Get raw video and perform preprocessing

1. Using the "ZegoVideoBufferTypeCVPixelBuffer" type video preprocessing method requires implementing the onCapturedUnprocessedCVPixelBuffer callback method. After the SDK obtains raw video data, it will notify the developer through the onCapturedUnprocessedCVPixelBuffer method.

2. Developers can use Express SDK's built-in basic beautification, watermark and other features to process video data. For details, please refer to Publishing Stream Video Enhancement; can also be used with Effects SDK to implement more processing effects. For details, please refer to Using Real-time Audio and Video with AI Beautification Together.

   Warning

   If developers use third-party beautification libraries or pre-allocate memory, please:

   • When custom video preprocessing starts (onStart), initialize tools and allocate memory.
   • When custom video preprocessing ends (onStop), deinitialize tools and release memory.

3. After processing is completed, call the sendCustomVideoProcessedCVPixelBuffer interface to send the processed video frame data to the Express SDK for publishing. Other clients can play the processed video stream.

- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
    // Custom preprocessing: Here use Effects SDK for video processing
    [self.effects processImageBuffer:buffer];

    // Send the processed buffer back to Express SDK
    [[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:buffer timestamp:timestamp channel:channel];
}

- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
    // Custom preprocessing: Here use Effects SDK for video processing
    [self.effects processImageBuffer:buffer];

    // Send the processed buffer back to Express SDK
    [[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:buffer timestamp:timestamp channel:channel];
}