Custom video pre-processing refers to processing the captured video with the AI Effects SDK for features such as face beautification, and stickers, this might be needed when the Video Call SDK does not meet your development requirements.
Compared to the custom video capture feature,
the custom video pre-processing does not require you to manage the device input sources, you only need to manipulate the raw data thrown out by the ZegoExpress-Video SDK
, and then send it back to the ZegoExpress-Video SDK
.
For more advanced features such as layer blending, we recommend you refer to the Customize how the video captures.
Before you begin, make sure you complete the following steps:
The following diagram shows the API call sequence of the custom video pre-processing:
ZegoCustomVideoProcessConfig
object, and set the bufferType
property for providing video frame data type to the SDK. Currently, the SDK only supports the CVPixelBuffer
type of video data (when the value of the bufferType
is ZegoVideoBufferTypeCVPixelBuffer
). Setting other enumeration values will not take effect.
enableCustomVideoProcessing
method before starting the local video preview and the stream publishing starts.ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select the [CVPixelBuffer] type of video frame data.
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;
[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];
Define a custom video pre-processing event handler object as ViewController
, and conforms to the ZegoCustomVideoProcessHandler
protocol.
@interface ViewController () <ZegoEventHandler, ZegoCustomVideoProcessHandler>
...
@end
Call the setCustomVideoProcessHandler
method to listen for and handle the custom video pre-processing event callbacks.
// Set [self] as the event callback object for custom video pre-processing.
[[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];
To get the video raw data (the data of the ZegoVideoBufferTypeCVPixelBuffer
type), call the onCapturedUnprocessedCVPixelBuffer
method.
After processed the raw data using the ZegoEffects SDK
, call the sendCustomVideoProcessedCVPixelBuffer
method to send the processed data.
- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
// Custom pre-processing: use the Effects SDK here.
[self.effects processImageBuffer:buffer];
// Send the processed buffer.
[[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:output timestamp:timestamp channel:channel];
}