Documentation
ExpressVideoSDK Video Call
Documentation
Demo APP
SDK Center
API Center
FAQ
Code Market
Console
Sign Up
Log In
中文站 English
  • Documentation
  • Video Call
  • Upgrade using advanced features
  • Distincitve features
  • Customize the video and audio
  • Pre-process the video

Pre-process the video

Last updated:2023-11-14 14:43

Introduction

Custom video pre-processing refers to processing the captured video with the AI Effects SDK for features such as face beautification, and stickers, this might be needed when the Video Call SDK does not meet your development requirements.

video_pre-processing

Compared to the custom video capture feature, the custom video pre-processing does not require you to manage the device input sources, you only need to manipulate the raw data thrown out by the ZegoExpress-Video SDK, and then send it back to the ZegoExpress-Video SDK.

For more advanced features such as layer blending, we recommend you refer to the Customize how the video captures.

Prerequisites

Before you begin, make sure you complete the following steps:

  • A project has been created in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - How to view project information .
  • ZEGO Express SDK has been integrated into the project to implement basic real-time audio and video functions. For details, please refer to Integrate the SDK and Implement a basic video call.

Implementation process

The following diagram shows the API call sequence of the custom video pre-processing:

Enable the custom video pre-processing feature

  1. Create a ZegoCustomVideoProcessConfig object, and set the bufferType property for providing video frame data type to the SDK.

Currently, the SDK only supports the CVPixelBuffer type of video data (when the value of the bufferType is ZegoVideoBufferTypeCVPixelBuffer). Setting other enumeration values will not take effect.

  1. To enable the custom video pre-processing, call the enableCustomVideoProcessing method before starting the local video preview and the stream publishing starts.
ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select the [CVPixelBuffer] type of video frame data.
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;

[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];

Define a event handler object

  1. Define a custom video pre-processing event handler object as ViewController, and conforms to the ZegoCustomVideoProcessHandler protocol.

    @interface ViewController () <ZegoEventHandler, ZegoCustomVideoProcessHandler>
    
        ...
    
    @end
  2. Call the setCustomVideoProcessHandler method to listen for and handle the custom video pre-processing event callbacks.

    // Set [self] as the event callback object for custom video pre-processing.
    [[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];

Get and process the video raw data

  1. To get the video raw data (the data of the ZegoVideoBufferTypeCVPixelBuffer type), call the onCapturedUnprocessedCVPixelBuffer method.

  2. After processed the raw data using the ZegoEffects SDK, call the sendCustomVideoProcessedCVPixelBuffer method to send the processed data.

- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
    // Custom pre-processing: use the Effects SDK here.
    [self.effects processImageBuffer:buffer];

    // Send the processed buffer.
    [[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:output timestamp:timestamp channel:channel];
}
Page Directory
  • Free trial
  • 提交工单
    咨询集成、功能及报价等问题
    电话咨询
    400 1006 604
    Get Consulting
    Scan Wechat QR code