Add video filters with the AI Effects

Introduction

The video call service provided by ZEGOCLOUD enables you to build audio and video applications through its flexible and easy-to-use API. Meanwhile, another ZEGOCLOUD add-on: AI Effects, is based on the AI algorithm, which enables you to implement a series of beautification features such as face beautification, face shape retouch, and other features.

These two services can be combined by using the two types of SDK together to create a real-time application with beautification features, which can be widely used in entertainment live streaming, live game streaming, video conference, and other live streaming scenarios.

Basic concepts

• The ZegoExpress-Video SDK (Hereafter called the Express SDK): The video call SDK provided by ZEGOCLOUD. This SDK enables you to implement real-time audio and video features, in live streaming, live co-hosting streaming, and other scenarios.

• The ZegoEffects SDK (Hereafter called the Effects SDK): The AI effects SDK provided by ZEGOCLOUD provides the AI-based image rendering and algorithm abilities that enable you to implement face beautification, face shape retouch, background segmentation, face detection, and other features.

Prerequisites

Before you begin, make sure you complete the following:

• Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.

• Refer to the Quick Start doc to complete the SDK integration and basic function implementation.

• Integrate the AI Effects SDK into your project. For details, see Integrate the SDK.
• Get the unique license file of the AI Effects SDK. For details, see Online authentication.

Implementation process

The overall process of the combination of the two SDKs is as follows:

1. Initialize the Express SDK and the Effects SDK (in no particular order).
2. Obtain the raw video data. For details, see Custom video capture or Custom video pre-processing.
3. Process the raw video data for beautification effects using the Effects SDK, and send the process video data, publish the video streams using the Express SDK.
4. To adjust the AI effects during the stream publishing and playing operation, you can use the related functions of the Effects SDK to make changes in real time.

Initialize the Effects SDK

Import the resources and models of the Effects SDK

Before using the AI features of the Effects SDK, you need to import the resources or models required for those features.

// Specify the absolute path of the face recognition model, which is required for features such as face detection, eyes enlarging, and face slimming.
ArrayList<String> aiModeInfos = new ArrayList<>();
aiModeInfos.add("sdcard/xxx/xxxxx/FaceDetectionModel.bundle");
aiModeInfos.add("sdcard/xxx/xxxxx/Segmentation.bundle");

// Specify the absolute path of the resources.
ArrayList<String> aiResources = new ArrayList<>();
aiResources.add("sdcard/xxx/xxxxx/PendantResources.bundle");
aiResources.add("sdcard/xxx/xxxxx/FaceWhiteningResources");

// Set the path list for resources or models, which must be called before calling the `create` method to create a ZegoEffects object.
ZegoEffects.setResources(aiResources);

Create an Effects SDK object

ZegoEffects mEffects;
long appid = *******;
String appSign = "*******";
ZegoEffects.create(appid, appSign, applicationContext, (effects, errorCode) -> {
    mEffects = effects;
    //Execute custom logic
});

// Pass in the authentication file you obtained.
ZegoEffects effects = ZegoEffects.create("ABCDEFG", getApplication());

Initialize the Effects SDK object

To initialize the Effects SDK object, call the initEnv method, and pass in the width and height of the incoming video data to be processed.

The following is an example of processing 1280 × 720 video images:

// Initialize the Effects SDK object, nd pass in the width and height of the incoming video data to be processed.
effects.initEnv(1280, 720);

Initialize the Express SDK

To initialize the Express SDK, call the createEngine method.

/** Define a ZegoExpressEngine object */
ZegoExpressEngine engine;

ZegoEngineProfile profile = new ZegoEngineProfile();
/** AppID format: 123456789L */
profile.appID = appID;
/** General scenario */
profile.scenario = ZegoScenario.GENERAL;
/** Set application object of App */
profile.application = getApplication();
/** Create a ZegoExpressEngine instance */
engine = ZegoExpressEngine.createEngine(profile, null);

Get the video raw data

The Express SDK provides two methods to get the video raw data:

• Custom video pre-processing: The Express SDK collects video raw data internally, and the video data can be obtained through a callback.
• Custom video capture: The video raw data is collected by developers themselves and then provided to the Express SDK.

The difference between the two methods is as follows, you can choose based on the actual situation:

• Method 1: Custom video pre-processing

To capture the video raw data using this method, do the following:

a. Select the GL_TEXTURE_2D video frame data type, and set the buffer type as RGBA32.

b. Call the enableCustomVideoProcessing method to enable the custom video pre-processing.

c. The SDK sends out the captured video raw data through the callback onCapturedUnprocessedTextureData.

ZegoCustomVideoProcessConfig config = new ZegoCustomVideoProcessConfig();
// Select the [GL_TEXTURE_2D] video frame data type.
config.bufferType = ZegoVideoBufferType.GL_TEXTURE_2D;

// Enable the custom video pre-processing.
express.enableCustomVideoProcessing(true, config, ZegoPublishChannel.MAIN);

For details, see Custom video pre-processing.

• Method 2: Custom video capture

For details, see Custom video capture.

Process the video data with the Effects SDK

In the corresponding callback for receiving the video raw data, the Effects SDK can be used to implement AI features.

After implemented the AI features, the Express SDK encodes and publishes the processed data and sends it to the cloud server. At this time, the remote user can play the processed video streams.

• If you capture the video data using the custom video pre-processing method, to process the captured video data for AI features, do the following:

1. Process the video data in the onCapturedUnprocessedTextureData callback.
2. After processed, send the video data back to the Express SDK.

// Take using the custom video pre-processing method as an example.
// Obtain the raw video data with callback.
// Set the callback.
express.setCustomVideoProcessHandler(new IZegoCustomVideoProcessHandler() {
    ...

    // Receive texture from ZegoExpressEngine
    @Override
    public void onCapturedUnprocessedTextureData(int textureID, int width, int height, long referenceTimeMillisecond, ZegoPublishChannel channel) {

        ZegoEffectsVideoFrameParam param = new ZegoEffectsVideoFrameParam();
        param.format = ZegoEffectsVideoFrameFormat.RGBA32;
        param.width = width;
        param.height = height;

        // Process buffer by ZegoEffects
        int processedTextureID = effects.processTexture(textureID, param);

        // Send processed texture to ZegoExpressEngine
        express.sendCustomVideoProcessedTextureData(processedTextureID, width, height, referenceTimeMillisecond);
    }
}

• If you capture the video data using the custom video capture method, to process the captured video data for AI features, do the following:

1. Process the video data in the corresponding callback (for details, see Custom video capture),
2. and send the processed data back to the Express SDK.

Adjust the beautification features using the Effects SDK

To adjust the AI effects during the stream publishing and playing operation, use the Effects SDK to make changes in real time.

// Enable the skin tone enhancement feature.
effects.enableWhiten(true);

// Set the whitening intensity. The value range is [0, 100], and the default value is 50.
ZegoEffectsWhitenParam param = new ZegoEffectsWhitenParam();
param.intensity = 100;
effects.setWhitenParam(param);

For more AI features, see Face beautification, Face shape retouch, Backgroud segmentation, Face detection, Stickers, and Filters.