Pre-process the video

---

Introduction

Custom video pre-processing refers to processing the captured video with the AI Effects SDK for features such as face beautification, and stickers, this might be needed when the Video Call SDK does not meet your development requirements.

Compared to the custom video capture feature, the custom video pre-processing does not require you to manage the device input sources, you only need to manipulate the raw data thrown out by the ZegoExpress-Video SDK, and then send it back to the ZegoExpress-Video SDK.

Prerequisites

Before you begin, make sure you complete the following:

• Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.

• Refer to the Quick Start doc to complete the SDK integration and basic function implementation.

Implementation process

The following diagram shows the API call sequence of the custom video pre-processing:

Enable the custom video pre-processing feature

Create a ZegoCustomVideoProcessConfig object and set the bufferType property to specify the type of video frame data provided to the Express SDK.

Currently, the Android SDK supports the following three bufferType values. Setting other enumeration values will not work properly:

Before starting the local video preview and stream publishing, call the enableCustomVideoProcessing method to enable the custom video pre-processing feature.

The following example demonstrates how to enable custom video pre-processing with ZegoVideoBufferType.GL_TEXTURE_2D:

ZegoCustomVideoProcessConfig config = new ZegoCustomVideoProcessConfig();
// Specify the GL_TEXTURE_2D type for video frame data.
config.bufferType = ZegoVideoBufferType.GL_TEXTURE_2D;

// Enable custom video pre-processing.
express.enableCustomVideoProcessing(true, config, ZegoPublishChannel.MAIN);

Get and process the raw video data

1. Call the setCustomVideoProcessHandler method to register a custom video pre-processing callback and implement the onCapturedUnprocessedTextureData and other related callback methods.

2. Once the SDK captures raw video frames, it notifies the developer through the onCapturedUnprocessedTextureData callback.

3. You can process the raw video frames using:

   • The basic enhancement features provided by the Express SDK, such as beauty filters and watermarks. For details, refer to Improve your appearance in the video.
   • The Effects SDK for more advanced video effects. For details, refer to Add video filters with the AI Effects.

4. After processing, call the sendCustomVideoProcessedTextureData method to send the processed video frame data back to the Express SDK for publishing. Other clients will then be able to play the processed video stream.

// Register the callback to retrieve and process raw data.
express.setCustomVideoProcessHandler(new IZegoCustomVideoProcessHandler() {
    ...

    // Receive the texture from the ZegoExpressEngine.
    @Override
    public void onCapturedUnprocessedTextureData(int textureID, int width, int height, long referenceTimeMillisecond, ZegoPublishChannel channel) {

        ZegoEffectsVideoFrameParam param = new ZegoEffectsVideoFrameParam();
        param.format = ZegoEffectsVideoFrameFormat.BGRA32;
        param.width = width;
        param.height = height;

        // Perform custom pre-processing using the Effects SDK.
        int processedTextureID = effects.processTexture(textureID, param);

        // Send the processed buffer back to the Express SDK.
        express.sendCustomVideoProcessedTextureData(processedTextureID, width, height, referenceTimeMillisecond);
    }
});