Object Segmentation
Feature Overview
Object Segmentation is a value-added capability provided by Express SDK. It uses AI algorithms to identify content in the video frame and sets transparency information for each pixel. Pixels belonging to the main subject are set as "opaque", while pixels outside the main subject are set as "transparent". Developers can use the transparency information of these pixels to apply different processing to the subject and non-subject parts of the frame, thereby implementing various features.
- The current official SDK does not include "Object Segmentation" related features. If needed, please contact ZEGOCLOUD Technical Support for special packaging and provide your AppID to enable the relevant permissions.
- The "Object Segmentation" feature is a paid feature. For trial applications or formal pricing inquiries, please contact ZEGOCLOUD sales representatives.
Object Segmentation Types
For users in different environments, ZEGO provides two segmentation capabilities: "Green Screen Background Segmentation" and "Arbitrary Background Segmentation".
| Segmentation Type | Green Screen Background Segmentation | Arbitrary Background Segmentation |
|---|---|---|
| Capability Description | When users have set up a green screen themselves, the subject in non-green screen areas can be retained. Suitable for e-commerce live streaming, online exams, and other scenarios. | Most users do not have the conditions to set up a green screen. Through ZEGO's arbitrary background segmentation capability, the subject in the frame can be identified without a green screen. Suitable for online education, video conferencing, and other scenarios. |
| Illustration |  |  |
Feature Scenarios
Based on the object segmentation capability, developers can implement business scenarios such as background blur, virtual background, presenter mode, and real-time multi-person interactive stages, creating more diverse interactive experiences.
| Feature | Background Blur | Virtual Background | Transparent Background | Subject Segmentation and Transmission |
|---|---|---|---|---|
| Feature Description | Apply blur processing to the area outside the subject. | Replace the area outside the subject with custom images, videos, or colors. | Render the subject on top of other video content locally. For example, implement presenter mode and other features on top of content such as screen sharing or playing videos. | Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented subject from the publishing end to the playing end, where subject rendering is performed to achieve the visual effect of multiple people in different locations appearing in the same scene in real time |
| Illustration |  |  |  |  |
Hardware Compatibility
| Platform | Hardware Requirements |
|---|---|
| Android |
|
| iOS | A series chips: Apple A9 and above, e.g., iPhone 6s |
| macOS | M series chips: Apple M1 and above |
| Windows | Intel Core i5 and above |
Sample Source Code Download
Please refer to Download Sample Source Code to obtain the source code.
For related source code, please check the files in the "/ZegoExpressExample/Others/src/main/java/com/example/others/videoobjectsegmentation" directory.
Prerequisites
Before implementing object segmentation features, please ensure:
-
You have contacted ZEGOCLOUD Technical Support for special packaging.
-
A project has been created in the ZEGOCLOUD Console, and valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.
-
ZEGO Express SDK has been integrated into the project, and basic audio/video publishing and playing functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
Implementation Steps
- Enabling object segmentation consumes additional system resources. To ensure user experience, currently only one channel's publishing stream can have object segmentation enabled.
- If custom third-party filters with pre-processing are used, ensure that the third-party filters support Alpha channel pass-through functionality.
Please note that developers can choose whether to implement the (Optional) steps in the above diagram based on their business scenario requirements. For implementation, please refer to the specific instructions below.
1 Initialization and Room Login
For specific steps on initialization and room login, please refer to "Create Engine" and "Login Room" in the implementing video call documentation.
2 (Optional) Implement Custom Rendering Class
Developers can perform custom rendering based on actual conditions. Note that the video raw data contains an Alpha channel, and the raw data has not undergone Alpha premultiplication processing. Taking rendering playing stream data as an example:
-
Listen for custom rendering data callbacks and cache rendering frame data.
WarningRendering cannot be done directly in the callback as it will block the callback thread, causing rendering abnormalities.
@Override void VideoObjectSegmentation::onRemoteVideoFrameRawData(unsigned char** data, unsigned int* dataLength, ZegoVideoFrameParam param, const std::string& streamID) { Q_UNUSED(dataLength) if(param.format == ZEGO_VIDEO_FRAME_FORMAT_BGRA32){ QImage image(data[0], param.width, param.height, param.strides[0], QImage::Format_ARGB32); if(image.isNull()) { // Image is empty, return return; } // Render in UI thread if (streamID == m_playStreamId1) { sigRenderPlayFrames1(image); } if (streamID == m_playStreamId2) { sigRenderPlayFrames2(image); } } } -
Draw the bitmap to the rendering view.
auto pix = QPixmap::fromImage(image); auto pixmap = pix.scaled(ui->label_playing_stream_1->width(), ui->label_playing_stream_1->height(),Qt::KeepAspectRatio); ui->label_playing_stream_1->setPixmap(pixmap);
3 (Optional) Enable Custom Rendering
Call the enableCustomVideoRender interface, set the engine advanced configuration and enable custom rendering, and call the setCustomVideoRenderHandler interface to set the custom video rendering callback. For details, please refer to Custom Video Rendering.
When using custom rendering for object segmentation, bufferType only supports the ZegoVideoBufferType.RAW_DATA type.
ZegoCustomVideoRenderConfig customVideoRenderConfig;
customVideoRenderConfig.bufferType = ZegoVideoBufferType::ZEGO_VIDEO_BUFFER_TYPE_RAW_DATA;// Select RAW_DATA type video frame data
customVideoRenderConfig.frameFormatSeries = ZEGO_VIDEO_FRAME_FORMAT_SERIES_RGB;// Select RGB color series data format
engine->enableCustomVideoRender(true, &customVideoRenderConfig);// Start custom video rendering
engine->setCustomVideoRenderHandler(std::make_shared<CustomVideoRenderer>(this));// Set custom video rendering callback4 Listen for Object Segmentation Status Callback
Call the onVideoObjectSegmentationStateChanged interface to listen for object segmentation status callbacks.
The object segmentation status callback depends on enabling preview or publishing stream. To listen for the onVideoObjectSegmentationStateChanged callback, you need to call preview startPreview or publish stream startPublishingStream.
void VideoObjectSegmentation::onVideoObjectSegmentationStateChanged(ZegoObjectSegmentationState state, ZegoPublishChannel channel, int errorCode)
{
if(state == ZEGO_OBJECT_SEGMENTATION_STATE_ON){
// Object segmentation enabled
}
else
{
// Object segmentation disabled
// Abnormal disable, please check the error code
}
}5 Use Object Segmentation to Implement Different Business Features
If developers need to update the object segmentation type or background processing type, they need to modify the ZegoObjectSegmentationConfig configuration and call the enableVideoObjectSegmentation interface again to enable object segmentation. The update result will be notified to developers through the onVideoObjectSegmentationStateChanged callback.
Background Blur
Call the enableVideoObjectSegmentation interface to enable object segmentation and set the background processing type to "Blur".
ZegoObjectSegmentationConfig config;
config.objectSegmentationType = ZEGO_OBJECT_SEGMENTATION_TYPE_ANY_BACKGROUND;// Select the required object segmentation type based on actual conditions
config.backgroundConfig.processType = ZEGO_BACKGROUND_PROCESS_TYPE_BLUR;// Set background processing method to blur
config.backgroundConfig.blurLevel = ZEGO_BACKGROUND_BLUR_LEVEL_MEDIUM;// Set background blur level to medium
engine->enableVideoObjectSegmentation(true, config, ZEGO_PUBLISH_CHANNEL_MAIN); // Enable object segmentationVirtual Background
Virtual background supports two types of materials:
- Images. Currently supports "PNG" and "JPEG" image formats, i.e., image files with ".png", ".jpg", and ".jpeg" extensions.
- Videos, with the following restrictions:
- Video format: MP4, FLV, MKV, AVI.
- Video source: Local video.
- Video playback mode: Loop playback.
- Resolution: Maximum not exceeding 4096 px, recommended within 1920 px.
- Video duration: Maximum not exceeding 30 seconds, recommended within 15 seconds.
- Video size: Maximum not exceeding 50 MB, recommended within 10 MB.
Developers using this feature should pay attention to the aspect ratio of custom images and video materials, otherwise portions exceeding the view will be cropped.
Call the enableVideoObjectSegmentation interface to enable object segmentation and set the background processing type to "Image" or "Video".
ZegoObjectSegmentationConfig config;
config.objectSegmentationType = ZEGO_OBJECT_SEGMENTATION_TYPE_ANY_BACKGROUND;// Select the required object segmentation type based on actual conditions
// Set background processing method to Image
config.backgroundConfig.processType = ZEGO_BACKGROUND_PROCESS_TYPE_IMAGE;
config.backgroundConfig.imageURL = "<image_path>";// Set background image path
engine->enableVideoObjectSegmentation(true, config, ZEGO_PUBLISH_CHANNEL_MAIN); // Enable object segmentation
// Set background processing method to Video
config.backgroundConfig.processType = ZEGO_BACKGROUND_PROCESS_TYPE_VIDEO;
config.backgroundConfig.videoURL = "<video_path>";// Set background video path
engine->enableVideoObjectSegmentation(true, config, ZEGO_PUBLISH_CHANNEL_MAIN); // Enable object segmentationTransparent Background
If developers need to implement business features similar to "Presenter Mode", they need to mix the "Subject Frame" with "Video Source Content to be Mixed" into a single stream on the business side.
Call the enableVideoObjectSegmentation interface to enable object segmentation and set the background processing type to "Transparent".
ZegoObjectSegmentationConfig config;
config.objectSegmentationType = ZEGO_OBJECT_SEGMENTATION_TYPE_ANY_BACKGROUND;// Select the required object segmentation type based on actual conditions
config.backgroundConfig.processType = ZEGO_BACKGROUND_PROCESS_TYPE_TRANSPARENT;// Set background processing method to transparent
engine->enableVideoObjectSegmentation(true, config, ZEGO_PUBLISH_CHANNEL_MAIN); // Enable object segmentation6 (Optional) Use Alpha Channel to Transmit Segmented Subject
If the publishing end needs to transmit the segmented subject frame to the playing end through the Alpha channel for subject rendering at the playing end, you need to first call the enableAlphaChannelVideoEncoder interface to set the encoder to support transparent channel, then call the startPublishingStream interface to publish the stream to successfully transmit it to the playing end.
Currently only supports transparent channel data arranged below RGB or YUV data.
-
Enable Alpha channel data transmission:
ZegoAlphaLayoutType layoutType = ZEGO_ALPHA_LAYOUT_TYPE_BOTTOM; // Transparent channel data arranged below RGB or YUV data engine->enableAlphaChannelVideoEncoder(true,layoutType, ZEGO_PUBLISH_CHANNEL_MAIN); // Set encoder to support transparent channel -
Disable Alpha channel data transmission:
ZegoAlphaLayoutType layoutType = ZEGO_ALPHA_LAYOUT_TYPE_BOTTOM; // Transparent channel data arranged below RGB or YUV data engine->enableAlphaChannelVideoEncoder(false,layoutType, ZEGO_PUBLISH_CHANNEL_MAIN); // Set encoder to support transparent channel
7 Start Preview and Publish Stream
After enabling object segmentation through the enableVideoObjectSegmentation interface, you can proceed with preview.
Developers can also enable preview first, then enable object segmentation. This document uses enabling object segmentation first, then preview as an example.
-
Method 1: Use internal rendering
If using internal rendering, before starting preview, please set alphaBlend to true.
ZegoCanvas canvas; canvas.view = ZegoUtilHelper::GetView(ui->label_preview); canvas.alphaBlend = true;// Enable internal rendering Alpha blending. After enabling, supports Alpha blending of segmented subject with background layer engine->startPreview(&canvas); engine->startPublishingStream(streamID); -
Method 2: Use custom rendering
If using custom rendering, you can directly proceed with preview and publish stream.
engine->startPreview(); engine->startPublishingStream(streamID);
8 (Optional) Set Alpha Channel Rendering at Playing End and Start Playing Stream
Alpha channel rendering needs to be enabled at the playing end only when the publishing end has enabled Alpha channel transmission.
-
Method 1: Use internal rendering
If using internal rendering, before calling the startPlayingStream interface to start playing stream, the playing end needs to set the alphaBlend property of ZegoCanvas to true.
ZegoCanvas canvas; canvas.view = ZegoUtilHelper::GetView(ui->label_playing_stream_1); canvas.alphaBlend = true;// Enable internal rendering Alpha blending. After enabling, supports Alpha blending of segmented subject with background layer engine->startPlayingStream(streamID, &canvas); -
Method 2: Use custom rendering
If using custom rendering, you can directly proceed with playing stream.
engine->startPlayingStream(streamID, nullptr);
9 Disable Object Segmentation
Call the enableVideoObjectSegmentation interface to disable object segmentation.
ZegoObjectSegmentationConfig config;
engine->enableVideoObjectSegmentation(false, config, ZEGO_PUBLISH_CHANNEL_MAIN);// Disable object segmentation10 Destroy Engine
Call the destroyEngine interface to destroy the engine.
ZegoExpressSDK::destroyEngine(engine);
engine = nullptr;