Object Segmentation

Feature Overview

Object Segmentation is a value-added capability provided by Express SDK. It uses AI algorithms to recognize content in video frames and sets transparency information for each pixel. Pixels of the main subject are set to "opaque", while pixels outside the main subject are set to "transparent". Developers can use the transparency information of these pixels to process the subject and non-subject parts of the image differently, thereby implementing different features.

Object Segmentation Types

For users in different environments, ZEGO provides two segmentation capabilities: "Green Screen Background Segmentation" and "Arbitrary Background Segmentation".

Segmentation Type	Green Screen Background Segmentation	Arbitrary Background Segmentation
Capability Description	When users set up a green screen themselves, the subject in the non-green screen area can be retained. Suitable for e-commerce live streaming, online exams, and other scenarios.	Most users do not have the conditions to set up a green screen. Through ZEGO's arbitrary background segmentation capability, the subject in the image can be recognized without a green screen. Suitable for online education, video conferences, and other scenarios.
Illustration

Feature Scenarios

Based on the object segmentation capability, developers can implement business scenarios such as background blurring, virtual backgrounds, presenter mode, and multi-user real-time interaction, creating more diverse interactive experiences.

Feature	Background Blur	Virtual Background	Transparent Background	Subject Segmentation and Transmission
Feature Description	Blur the area outside the subject.	Replace the area outside the subject with custom images, videos, or colors.	Render the subject's image over other video content locally. For example, implement features such as presenter mode over screen sharing or playing video content.	Combined with the Alpha channel data transmission capability provided by Express SDK, the segmented subject is transmitted to the playing end and rendered at the playing end, achieving the visual effect of multiple people in different locations appearing in the same scene in real time
Illustration

Hardware Compatibility

Please note, when using Flutter SDK on the above platforms, hardware compatibility is as above.

Sample Source Code

Please refer to Download Sample Source Code to get the source code.

For related source code, please check files in the "lib/topics/OtherFunctions/screen_sharing" directory.

Prerequisites

Before using the object segmentation feature, ensure:

• You have contacted ZEGOCLOUD Technical Support for a special build.

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.

• You have integrated ZEGO Express SDK in your project and implemented basic audio and video publishing and playing features. For details, please refer to Quick Start - Integrating SDK and Quick Start - Implementing Video Call.

Implementation

Please note, developers can choose whether to implement the (Optional) steps in the above diagram according to their business scenario needs. If implementation is needed, please refer to the specific instructions below.

Initialization and Room Login

For the specific process of initialization and room login, please refer to "Create Engine" and "Login Room" in the Implementing Video Call documentation.

Listen for Object Segmentation State Callback

Call the onVideoObjectSegmentationStateChanged interface to listen for object segmentation state callbacks.

ZegoExpressEngine.onVideoObjectSegmentationStateChanged = (state, channel, errorCode) {
  ZegoLog().addLog(
      'Video object segmentation state changed. state: $state, errorCode: $errorCode, channel: $channel');
};

ZegoExpressEngine.onVideoObjectSegmentationStateChanged = (state, channel, errorCode) {
  ZegoLog().addLog(
      'Video object segmentation state changed. state: $state, errorCode: $errorCode, channel: $channel');
};

Use Object Segmentation to Implement Different Business Features

Background Blur

ZegoObjectSegmentationConfig config = ZegoObjectSegmentationConfig.defaultConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Blur;//Set background processing method to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevel.Medium;//Set background blur level to medium
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

ZegoObjectSegmentationConfig config = ZegoObjectSegmentationConfig.defaultConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Blur;//Set background processing method to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevel.Medium;//Set background blur level to medium
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

Virtual Background

ZegoObjectSegmentationConfig config = ZegoObjectSegmentationConfig.defaultConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to enable based on actual situation

//Set background processing method to Image
config.backgroundConfig.processType = ZegoBackgroundProcessType.Image;
config.backgroundConfig.imageURL = "<image_path>";//Set background image path
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

//Set background processing method to Video
config.backgroundConfig.processType = ZegoBackgroundProcessType.Video;
config.backgroundConfig.videoURL = "<video_path>";//Set background video path
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

ZegoObjectSegmentationConfig config = ZegoObjectSegmentationConfig.defaultConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to enable based on actual situation

//Set background processing method to Image
config.backgroundConfig.processType = ZegoBackgroundProcessType.Image;
config.backgroundConfig.imageURL = "<image_path>";//Set background image path
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

//Set background processing method to Video
config.backgroundConfig.processType = ZegoBackgroundProcessType.Video;
config.backgroundConfig.videoURL = "<video_path>";//Set background video path
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

Transparent Background

ZegoObjectSegmentationConfig config = ZegoObjectSegmentationConfig.defaultConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Transparent;//Set background processing method to transparent
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

ZegoObjectSegmentationConfig config = ZegoObjectSegmentationConfig.defaultConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.Transparent;//Set background processing method to transparent
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.Main);//Enable object segmentation

(Optional) Use Alpha Channel to Transmit Segmented Subject

Start Preview and Publishing

After enabling the object segmentation feature through the enableVideoObjectSegmentation interface, you can preview.

ZegoCanvas canvas = ZegoCanvas(previewView);
canvas.alphaBlend = true;//Enable internal rendering Alpha blending. After enabling, it supports Alpha blending between the segmented subject and background layer
engine.startPreview(canvas);
engine.startPublishingStream(streamID);

ZegoCanvas canvas = ZegoCanvas(previewView);
canvas.alphaBlend = true;//Enable internal rendering Alpha blending. After enabling, it supports Alpha blending between the segmented subject and background layer
engine.startPreview(canvas);
engine.startPublishingStream(streamID);

(Optional) Set Alpha Channel Rendering at Playing End and Start Playing

ZegoCanvas canvas = ZegoCanvas(playView);
canvas.alphaBlend = true;//Enable internal rendering Alpha blending. After enabling, it supports Alpha blending between the segmented subject and background layer
engine.startPlayingStream(streamID, canvas);

ZegoCanvas canvas = ZegoCanvas(playView);
canvas.alphaBlend = true;//Enable internal rendering Alpha blending. After enabling, it supports Alpha blending between the segmented subject and background layer
engine.startPlayingStream(streamID, canvas);

Disable Object Segmentation

Call the enableVideoObjectSegmentation interface to disable object segmentation.

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to disable based on actual situation
engine.enableVideoObjectSegmentation(false, objectType, ZegoPublishChannel.Main);//Disable object segmentation

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationType.AnyBackground;//Select the object segmentation type to disable based on actual situation
engine.enableVideoObjectSegmentation(false, objectType, ZegoPublishChannel.Main);//Disable object segmentation

Destroy Engine

Call the destroyEngine interface to destroy the engine.

ZegoExpressEngine.destroyEngine();

ZegoExpressEngine.destroyEngine();