Pre-process the video

Introduction

Custom video pre-processing refers to processing the captured video with the AI Effects SDK for features such as face beautification, and stickers, this might be needed when the Video Call SDK does not meet your development requirements.

Compared to the custom video capture feature, the custom video pre-processing does not require you to manage the device input sources, you only need to manipulate the raw data thrown out by the ZegoExpress-Video SDK, and then send it back to the ZegoExpress-Video SDK.

Prerequisites

Before you begin, make sure you complete the following steps:

• ZEGO Express SDK has been integrated into the project to implement basic real-time audio and video functions. For details, please refer to Quick start .
• A project has been created in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information .

Implementation process

The following diagram shows the API call sequence of the custom video pre-processing:

Enable the custom video pre-processing feature

1. Create a ZegoCustomVideoProcessConfig object, and set the bufferType property for providing video frame data type to the SDK.

2. To enable the custom video pre-processing, call the enableCustomVideoProcessing method before starting the local video preview and the stream publishing starts.

ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select the [CVPixelBuffer] type of video frame data.
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;

[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];

Define a event handler object

1. Define a custom video pre-processing event handler object as ViewController, and conforms to the ZegoCustomVideoProcessHandler protocol.

   @interface ViewController () <ZegoEventHandler, ZegoCustomVideoProcessHandler>
   
       ...
   
   @end

2. Call the setCustomVideoProcessHandler method to listen for and handle the custom video pre-processing event callbacks.

   // Set [self] as the event callback object for custom video pre-processing.
   [[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];

Get and process the video raw data

1. To get the video raw data (the data of the ZegoVideoBufferTypeCVPixelBuffer type), call the onCapturedUnprocessedCVPixelBuffer method.

2. After processed the raw data using the ZegoEffects SDK, call the sendCustomVideoProcessedCVPixelBuffer method to send the processed data.

- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
    // Custom pre-processing: use the Effects SDK here.
    [self.effects processImageBuffer:buffer];

    // Send the processed buffer.
    [[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:output timestamp:timestamp channel:channel];
}