Custom Video Preprocessing
Feature Overview
Video preprocessing is a process between capture and encoding. Developers can perform video preprocessing through the SDK's built-in basic AI effects and watermark features after capturing video data themselves or obtaining video data captured by the SDK. If the SDK cannot meet the developers' needs (for example, the beauty effect cannot achieve the expected results), they can also use the ZEGO Effects SDK to perform special processing on the video, such as beauty effects, adding accessories, etc. This process is called custom video preprocessing.
Compared with custom video capture, the advantage of custom video preprocessing is that developers do not need to manage device input sources, but only need to operate on the original data provided by ZEGO Express SDK, and then send it back to ZEGO Express SDK.
For more complex scenarios, such as wanting to perform layer blending through camera images, it is recommended that developers use the custom video capture feature to implement it. This method has more room for performance optimization.
Download Example Source Code
ZEGO provides example source code for implementing beauty effects through the Effects SDK. Please refer to Using Video Call and AI Effects Together.
Prerequisites
Before performing custom video preprocessing, ensure that:
- A project has been created in the ZEGOCLOUD Console and valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.
- ZEGO Express SDK has been integrated into your project and basic audio and video streaming functions have been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
Usage Steps
The flow and interface calls for custom video preprocessing are as follows:
-
Initialize Express SDK and log in to the room.
-
Call the enableCustomVideoProcessing interface to enable custom video preprocessing.
-
Obtain original video data and perform video preprocessing.
- Call the setCustomVideoProcessHandler interface to register the custom video preprocessing callback and implement callback methods such as onCapturedUnprocessedCVPixelBuffer.
- Start preview and publish stream.
- Obtain original video data through the onCapturedUnprocessedCVPixelBuffer callback interface. Developers can perform related processing according to business needs: they can process through Express SDK's built-in basic AI effects, watermark and other functions; they can also use Effects SDK to achieve more processing effects.
- After the video data processing is completed, call the sendCustomVideoProcessedCVPixelBuffer interface in the onCapturedUnprocessedCVPixelBuffer callback to send the processed video frame data to Express SDK.
-
Other clients can play the processed video stream.
-
After completion, stop preview and publishing stream, leave the room, destroy the engine, and release resources.
1 Enable Custom Video Preprocessing
Create a custom video preprocessing ZegoCustomVideoProcessConfig object and set the bufferType property to provide the video frame data type.
Currently, the macOS SDK supports the following 1 bufferType data type. Setting other enumeration values will not work properly:
| Buffer Type | Enumeration Value | Description |
|---|---|---|
| CVPixelBuffer | ZEGO_VIDEO_BUFFER_TYPE_CV_PIXEL_BUFFER | Indicates original video data of CVPixelBufferRef type, format is BGRA32. |
Before starting preview and publishing stream, call the enableCustomVideoProcessing interface to enable the custom video preprocessing feature.
ZegoCustomVideoProcessConfig config;
// Select CVPixelBuffer type video frame data
config.bufferType = ZEGO_VIDEO_BUFFER_TYPE_CV_PIXEL_BUFFER;
engine->enableCustomVideoProcessing(true,&config);2 Obtain Original Video Data and Perform Video Preprocessing
Register Custom Video Preprocessing Callback
-
Use "CustomVideoProcessHandler" as the custom video preprocessing callback object, following the IZegoCustomVideoProcessHandler protocol.
class CustomVideoProcessHandler : public IZegoCustomVideoProcessHandler { ...... }; -
Call the setCustomVideoProcessHandler interface to set the custom video capture callback.
engine->setCustomVideoProcessHandler(customVideoProcessHandler);
Obtain Original Video and Perform Preprocessing
-
Using the "ZEGO_VIDEO_BUFFER_TYPE_CV_PIXEL_BUFFER" type video preprocessing method requires implementing the onCapturedUnprocessedCVPixelBuffer callback method. When the SDK obtains original video data, it will notify the developer through the onCapturedUnprocessedCVPixelBuffer method.
-
Developers can use Express SDK's built-in basic AI effects, watermark and other functions to process video data. For details, please refer to Publish Video Enhancement; they can also use Effects SDK to achieve more processing effects. For details, please refer to Using Video Call and AI Effects Together.
-
After processing is completed, call the sendCustomVideoProcessedCVPixelBuffer interface to send the processed video frame data to Express SDK for publishing, and other clients can play the processed video stream.
void CustomVideoProcessHandler::onCapturedUnprocessedCVPixelBuffer(void * buffer, unsigned long long referenceTimeMillisecond, ZegoPublishChannel channel) {
CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)buffer;
CVReturn cvRet = CVPixelBufferLockBaseAddress(pixelBuffer, 0);
if (cvRet != kCVReturnSuccess) return;
int width = CVPixelBufferGetWidth(pixelBuffer);
int height = CVPixelBufferGetHeight(pixelBuffer);
int stride = CVPixelBufferGetBytesPerRow(pixelBuffer);
unsigned char *dest = (unsigned char *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
QImage image(dest,width,height,stride,QImage::Format_ARGB32);
zego_effects_video_frame_param param;
param.format = zego_effects_video_frame_format_bgra32;
param.width = image.width();
param.height = image.height();
// Custom preprocessing: Use ZegoEffects SDK for video processing here
zego_effects_process_image_buffer_rgb(m_handle,image.bits(), image.bytesPerLine() * image.height(),param);
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
// Send the processed buffer back to ZegoExpress-Video SDK
engine->sendCustomVideoProcessedCVPixelBuffer(buffer,referenceTimeMillisecond,channel);
}