Object Segmentation

Feature Introduction

Object segmentation is a value-added capability provided by Express SDK. It uses AI algorithms to recognize content in video images and sets transparency information for each pixel. Among them, the pixels of the main subject will be set to "opaque", and the pixels outside the main subject will be set to "transparent". Developers can use the transparency information of these pixels to process the main subject and non-main subject parts in the image differently, thereby realizing different functions.

Object Segmentation Objects

For users in different environments, ZEGO provides two segmentation capabilities: "Green Screen Background Segmentation" and "Arbitrary Background Segmentation".

Segmentation Type	Green Screen Background Segmentation	Arbitrary Background Segmentation
Capability Description	When users set up a green screen by themselves, the main subject in the non-green screen area can be retained. Suitable for e-commerce live streaming, online exams and other scenarios.	Most users do not have the conditions to set up a green screen. Through the arbitrary background segmentation capability provided by ZEGO, the main subject in the image can be recognized without a green screen. Suitable for online education, video conferences and other scenarios.
Illustration

Functional Scenarios

Based on the object segmentation capability, developers can implement business scenarios such as background blurring, virtual background, presentation mode, and multi-person real-time co-location interaction, creating more diverse interactive experiences.

Function Point	Background Blurring	Virtual Background	Background Transparency	Object Segmentation and Transmission
Function Description	Blur the image outside the main subject.	Replace the image outside the main subject with custom images and colors.	Render the image of the main subject on other video content locally. For example, on content such as screen sharing or currently playing videos, implement functions such as presentation mode.	Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented main subject in the image to the playing end, and render the main subject at the playing end to achieve the visual effect of multiple people in different places appearing in the same scene in real time
Illustration

Hardware Compatibility

Example Source Code

Please refer to Download Example Source Code to obtain the source code.

For related source code, please check the files in the "/ZegoExpressExample/Others/src/main/java/com/example/others/videoobjectsegmentation" directory.

Prerequisites

Before using the object segmentation function, please ensure:

• You have contacted ZEGOCLOUD Technical Support for special packaging.

• A project has been created in the ZEGOCLOUD Console, and valid AppID and AppSign have been applied for. For details, please refer to Console - Project Information.

• ZEGO Express SDK has been integrated into the project, and basic audio and video publishing and playing functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation Process.

Implementation Process

Please note that developers can choose whether to implement the (Optional) steps in the above figure according to their business scenario needs. If implementation is needed, please refer to the specific instructions below.

Initialize and Login to Room

For the specific process of initialization and logging in to the room, please refer to "Create Engine" and "Login Room" in the Implement Video Call document.

Listen to Object Segmentation State Callback

Call the onVideoObjectSegmentationStateChanged interface to listen to the object segmentation state callback.

zgEngine.on('onVideoObjectSegmentationStateChanged', (res) =>{
   console.log('onVideoObjectSegmentationStateChanged: '+ res.state)
   if(res.state == zgDefines.ZegoObjectSegmentationState.On)
   {
       //Object segmentation enabled
   }
   else
   {
      //Object segmentation disabled
      //Abnormal shutdown, please check the error code
   }
})

zgEngine.on('onVideoObjectSegmentationStateChanged', (res) =>{
   console.log('onVideoObjectSegmentationStateChanged: '+ res.state)
   if(res.state == zgDefines.ZegoObjectSegmentationState.On)
   {
       //Object segmentation enabled
   }
   else
   {
      //Object segmentation disabled
      //Abnormal shutdown, please check the error code
   }
})

Use Object Segmentation to Implement Different Business Functions

Background Blurring

// Set background processing method to blur
// Set background blur level to medium
var config = {objectSegmentationType: zgDefines.ZegoObjectSegmentationType.AnyBackground, backgroundConfig: {processType: zgDefines.ZegoBackgroundProcessType.Blur, blurLevel: zgDefines.ZegoBackgroundBlurLevel.Medium}};
//Enable object segmentation
zgEngine.enableVideoObjectSegmentation(true, config, zgDefines.ZegoPublishChannel.Main)

// Set background processing method to blur
// Set background blur level to medium
var config = {objectSegmentationType: zgDefines.ZegoObjectSegmentationType.AnyBackground, backgroundConfig: {processType: zgDefines.ZegoBackgroundProcessType.Blur, blurLevel: zgDefines.ZegoBackgroundBlurLevel.Medium}};
//Enable object segmentation
zgEngine.enableVideoObjectSegmentation(true, config, zgDefines.ZegoPublishChannel.Main)

Virtual Background

// Set background processing method to image
var config = {objectSegmentationType: zgDefines.ZegoObjectSegmentationType.AnyBackground, backgroundConfig: {processType: zgDefines.ZegoBackgroundProcessType.Image, imageURL: "Set background image path"}};
//Enable object segmentation
zgEngine.enableVideoObjectSegmentation(true, config, zgDefines.ZegoPublishChannel.Main)

// Set background processing method to image
var config = {objectSegmentationType: zgDefines.ZegoObjectSegmentationType.AnyBackground, backgroundConfig: {processType: zgDefines.ZegoBackgroundProcessType.Image, imageURL: "Set background image path"}};
//Enable object segmentation
zgEngine.enableVideoObjectSegmentation(true, config, zgDefines.ZegoPublishChannel.Main)

Transparent Background

// Set background processing method to transparent
var config = {objectSegmentationType: zgDefines.ZegoObjectSegmentationType.AnyBackground, backgroundConfig: {processType: zgDefines.ZegoBackgroundProcessType.Transparent}};
//Enable object segmentation
zgEngine.enableVideoObjectSegmentation(true, config, zgDefines.ZegoPublishChannel.Main)

// Set background processing method to transparent
var config = {objectSegmentationType: zgDefines.ZegoObjectSegmentationType.AnyBackground, backgroundConfig: {processType: zgDefines.ZegoBackgroundProcessType.Transparent}};
//Enable object segmentation
zgEngine.enableVideoObjectSegmentation(true, config, zgDefines.ZegoPublishChannel.Main)

(Optional) Use Alpha Channel to Transmit Segmented Main Subject

Start Preview and Publishing Stream

After enabling the object segmentation function through the enableVideoObjectSegmentation interface, you can preview.

zgEngine.startPreview();
zgEngine.startPublishingStream(streamID);

zgEngine.startPreview();
zgEngine.startPublishingStream(streamID);

Play Stream

Start playing stream through the startPlayingStream interface.

zgEngine.startPlayingStream(StreamID, {
    canvas: localCanvas
});

zgEngine.startPlayingStream(StreamID, {
    canvas: localCanvas
});

Disable Object Segmentation

Call the enableVideoObjectSegmentation interface to disable object segmentation.

// Disable object segmentation
zgEngine.enableVideoObjectSegmentation(false, {}, zgDefines.ZegoPublishChannel.Main)

// Disable object segmentation
zgEngine.enableVideoObjectSegmentation(false, {}, zgDefines.ZegoPublishChannel.Main)

Destroy Engine

Call the destroyEngine interface to destroy the engine.

zgEngine.destroyEngine();

zgEngine.destroyEngine();