Custom Video Preprocessing

Feature Overview

Video preprocessing is a process between capture and encoding. Developers capture video data themselves. After obtaining the video data captured by the SDK, they can perform video preprocessing through the SDK's built-in basic beauty effects and watermark features. If the SDK cannot meet developers' needs (for example, the beauty effects cannot achieve the expected results), they can also combine with ZEGO Effects SDK to perform special processing on the video, such as beauty effects, adding accessories, etc. This process is custom video preprocessing.

Compared with custom video capture, custom video preprocessing has the advantage that developers do not need to manage device input sources, they only need to operate on the raw data provided by ZEGO Express SDK, and then send it back to ZEGO Express SDK.

Example Source Code Download

ZEGO provides example source code for implementing beauty effects through the Effects SDK. Please refer to Using Real-Time Video and AI Beauty Effects Together.

Prerequisites

Before performing custom video preprocessing, please ensure:

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
• You have integrated ZEGO Express SDK in your project and implemented basic audio and video streaming functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

The flow and interface calls for custom video preprocessing are as follows:

1 Enable Custom Video Preprocessing

Create a custom video preprocessing ZegoCustomVideoProcessConfig object and set the bufferType property to provide the video frame data type to Express SDK.

Currently Android SDK supports the following 3 types of bufferType data types. Setting other enumeration values will not work properly:

Before starting preview and starting to publish stream, call the enableCustomVideoProcessing interface to enable the custom video preprocessing feature.

The following takes "ZegoVideoBufferType.GL_TEXTURE_2D" as an example to demonstrate the usage of custom video preprocessing.

ZegoCustomVideoProcessConfig config = new ZegoCustomVideoProcessConfig();
// Select GL_TEXTURE_2D type video frame data
config.bufferType = ZegoVideoBufferType.GL_TEXTURE_2D;

// Enable custom preprocessing
express.enableCustomVideoProcessing(true, config, ZegoPublishChannel.MAIN);

ZegoCustomVideoProcessConfig config = new ZegoCustomVideoProcessConfig();
// Select GL_TEXTURE_2D type video frame data
config.bufferType = ZegoVideoBufferType.GL_TEXTURE_2D;

// Enable custom preprocessing
express.enableCustomVideoProcessing(true, config, ZegoPublishChannel.MAIN);

2 Get Raw Video Data and Perform Video Preprocessing

1. Call the setCustomVideoProcessHandler interface to register custom video preprocessing callbacks and implement callback methods such as onCapturedUnprocessedTextureData.

2. When the SDK obtains raw video data, it will notify the developer through the onCapturedUnprocessedTextureData method.

3. Developers can use Express SDK's built-in basic beauty effects, watermarks and other features to process video data. For details, please refer to Publish Stream Video Enhancement; they can also combine with Effects SDK to achieve more processing effects. For details, please refer to Using Real-Time Video and AI Beauty Effects Together.

   Warning

   If developers use third-party beauty libraries or pre-allocate memory, please:

   • Initialize tools and allocate memory when custom video preprocessing starts (onStart).
   • Deinitialize tools and release memory when custom video preprocessing ends (onStop).

4. After processing is completed, call the sendCustomVideoProcessedTextureData interface to send the processed video frame data to Express SDK, which will be published to the stream, and other clients can play the processed video stream.

// Callback method to get raw data
// Callback processing
express.setCustomVideoProcessHandler(new IZegoCustomVideoProcessHandler() {
    ...

    // Receive Texture from ZegoExpressEngine engine
    @Override
    public void onCapturedUnprocessedTextureData(int textureID, int width, int height, long referenceTimeMillisecond, ZegoPublishChannel channel) {


        ZegoEffectsVideoFrameParam param = new ZegoEffectsVideoFrameParam();
        param.format = ZegoEffectsVideoFrameFormat.BGRA32;
        param.width = width;
        param.height = height;

        // Custom preprocessing: Use Effects SDK for video processing here
        int processedTextureID = effects.processTexture(textureID, param);

        // Send the processed buffer back to Express SDK
        express.sendCustomVideoProcessedTextureData(processedTextureID, width, height, referenceTimeMillisecond);
    }
}

// Callback method to get raw data
// Callback processing
express.setCustomVideoProcessHandler(new IZegoCustomVideoProcessHandler() {
    ...

    // Receive Texture from ZegoExpressEngine engine
    @Override
    public void onCapturedUnprocessedTextureData(int textureID, int width, int height, long referenceTimeMillisecond, ZegoPublishChannel channel) {


        ZegoEffectsVideoFrameParam param = new ZegoEffectsVideoFrameParam();
        param.format = ZegoEffectsVideoFrameFormat.BGRA32;
        param.width = width;
        param.height = height;

        // Custom preprocessing: Use Effects SDK for video processing here
        int processedTextureID = effects.processTexture(textureID, param);

        // Send the processed buffer back to Express SDK
        express.sendCustomVideoProcessedTextureData(processedTextureID, width, height, referenceTimeMillisecond);
    }
}