Object Segmentation

Feature Overview

Object Segmentation is a value-added capability provided by the Express SDK. It uses AI algorithms to identify content in video frames and set transparency information for each pixel. Pixels in the main subject area are set to "opaque", while pixels outside the main subject area are set to "transparent". Developers can use this transparency information to apply different processing to the main subject and background areas, enabling various features.

Object Segmentation Types

For users in different environments, ZEGO provides two segmentation capabilities: "Green Screen Background Segmentation" and "Arbitrary Background Segmentation".

Segmentation Type	Green Screen Background Segmentation	Arbitrary Background Segmentation
Capability Description	When users have set up a green screen, the main subject in non-green screen areas can be retained. Suitable for e-commerce live streaming, online exams, and other scenarios.	Most users don't have the conditions to set up a green screen. Through ZEGO's arbitrary background segmentation capability, the main subject in the frame can be identified without a green screen. Suitable for online education, video conferences, and other scenarios.
Illustration

Feature Scenarios

Based on the Object Segmentation capability, developers can implement scenarios such as background blur, virtual background, presenter mode, and multi-user real-time interaction, creating more diverse interactive experiences.

Feature	Background Blur	Virtual Background	Transparent Background	Object Segmentation and Transmission
Feature Description	Blur the area outside the main subject.	Replace the area outside the main subject with custom images or colors.	Render the main subject over other video content locally. For example, implement presenter mode over screen sharing or playing video content.	Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented main subject to the playing stream side, where subject rendering is performed, achieving the visual effect of multiple users in different locations appearing in the same scene in real-time.
Illustration

Hardware Compatibility

Please note that when using ReactNative SDK on the above platforms, hardware compatibility requirements are the same.

Prerequisites

Before using the Object Segmentation feature, ensure that:

• You have contacted ZEGOCLOUD Technical Support for a special build.

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, refer to Console - Project Information.

• You have integrated ZEGO Express SDK in your project and implemented basic audio/video stream publishing and playing. For details, refer to Quick Start - Integration and Quick Start - Implementation.

Implementation Flow

Note that developers can choose whether to implement the (Optional) steps shown in the diagram above based on their business scenario needs. For implementation details, refer to the specific instructions below.

Initialization and Room Login

For the specific flow of initialization and room login, refer to "Create Engine" and "Login Room" in the Implementing Video Call documentation.

Listen for Object Segmentation State Callback

Call the videoObjectSegmentationStateChanged interface to listen for Object Segmentation state callbacks.

ZegoExpressEngine.instance().on('videoObjectSegmentationStateChanged', (state, channel, errorCdoe) => {
    console.log("JS videoObjectSegmentationStateChanged: " + state + " channel: " + channel + " errorCode: " + errorCdoe);
})

ZegoExpressEngine.instance().on('videoObjectSegmentationStateChanged', (state, channel, errorCdoe) => {
    console.log("JS videoObjectSegmentationStateChanged: " + state + " channel: " + channel + " errorCode: " + errorCdoe);
})

Use Object Segmentation to Implement Different Business Features

Background Blur

let config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Blur;//Set background processing mode to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevel.Medium;//Set background blur level to medium
ZegoExpressEngine.instance().enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable Object Segmentation

let config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Blur;//Set background processing mode to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevel.Medium;//Set background blur level to medium
ZegoExpressEngine.instance().enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable Object Segmentation

Virtual Background

let config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Image;//Set background processing mode to image
config.backgroundConfig.imageURL = "<image_path>";//Set background image path
ZegoExpressEngine.instance().enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable Object Segmentation

let config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Image;//Set background processing mode to image
config.backgroundConfig.imageURL = "<image_path>";//Set background image path
ZegoExpressEngine.instance().enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable Object Segmentation

Transparent Background

let config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Transparent;//Set background processing mode to transparent
ZegoExpressEngine.instance().enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable Object Segmentation

let config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Transparent;//Set background processing mode to transparent
ZegoExpressEngine.instance().enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable Object Segmentation

(Optional) Use Alpha Channel to Transmit Segmented Subject

Start Preview and Publish Stream

After enabling the Object Segmentation feature through the enableVideoObjectSegmentation interface, you can start preview.

let view = {"reactTag": findNodeHandle(this.refs.zego_preview_view), "viewMode": 0, "backgroundColor": 0};
view.alphaBlend = true;//Enable internal rendering Alpha blending, which supports Alpha blending between the segmented subject and background layer
ZegoExpressEngine.instance().startPreview(view);
ZegoExpressEngine.instance().startPublishingStream(streamID);

let view = {"reactTag": findNodeHandle(this.refs.zego_preview_view), "viewMode": 0, "backgroundColor": 0};
view.alphaBlend = true;//Enable internal rendering Alpha blending, which supports Alpha blending between the segmented subject and background layer
ZegoExpressEngine.instance().startPreview(view);
ZegoExpressEngine.instance().startPublishingStream(streamID);

(Optional) Set Alpha Channel Rendering on Playing Stream Side and Start Playing Stream

let view = {"reactTag": findNodeHandle(this.refs.zego_play_view), "viewMode": 0, "backgroundColor": 0};
view.alphaBlend = true;//Enable internal rendering Alpha blending, which supports Alpha blending between the segmented subject and background layer
ZegoExpressEngine.instance().startPlayingStream(streamID, canvas);

let view = {"reactTag": findNodeHandle(this.refs.zego_play_view), "viewMode": 0, "backgroundColor": 0};
view.alphaBlend = true;//Enable internal rendering Alpha blending, which supports Alpha blending between the segmented subject and background layer
ZegoExpressEngine.instance().startPlayingStream(streamID, canvas);

Disable Object Segmentation

Call the enableVideoObjectSegmentation interface to disable Object Segmentation.

let objectType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to disable based on actual situation
ZegoExpressEngine.instance().enableVideoObjectSegmentation(false, objectType, ZegoPublishChannel.Main);//Disable Object Segmentation

let objectType = ZegoObjectSegmentationType.AnyBackground;//Select the Object Segmentation type to disable based on actual situation
ZegoExpressEngine.instance().enableVideoObjectSegmentation(false, objectType, ZegoPublishChannel.Main);//Disable Object Segmentation

Destroy Engine

Call the destroyEngine interface to destroy the engine.

ZegoExpressEngine.destroyEngine();

ZegoExpressEngine.destroyEngine();