logo
ConsoleSign Up
Document
Client API
Server API
Video Call
Android: Java
Introduction
Overview
Product Features
Performance Data
Differences between Video Call SDK and Voice Call SDK
Client SDK
Download SDK
Experience App
Release Notes
Upgrade Guide
Common Error Codes
Quick Start
Example Source Code Running Guide
Integrating the SDK
Implement Video Call
Scenario-based Audio and Video Configuration
Connect to ZEGOCLOUD Document MCP Serverlink
Communication
Room Features
Audio
Video
Live Streaming
Other Features
Best Practices
FAQ
On this page

Object Segmentation

2024-09-06

Feature Overview

Object segmentation is a value-added capability provided by Express SDK. It uses AI algorithms to identify content in video frames and sets transparency information for each pixel. Among them, pixels in the subject part are set to "opaque", while pixels outside the subject part are set to "transparent". Developers can use the transparency information of these pixels to perform different processing on the subject and non-subject parts in the image, thereby achieving different functions.

Warning
  • The current official website SDK does not include "object segmentation" related functions. If needed, please contact ZEGO Technical Support for special packaging and provide your AppID to activate relevant permissions.
  • "Object segmentation" is a paid feature. If you need to apply for a trial or inquire about formal pricing, please contact ZEGO business personnel.

Object Segmentation Objects

For users in different environments, ZEGO provides two segmentation capabilities: "green screen background segmentation" and "arbitrary background segmentation".

Segmentation TypeGreen Screen Background SegmentationArbitrary Background Segmentation
Capability Description

When users set up a green screen themselves, the subject in the non-green screen area can be retained.

Suitable for e-commerce live streaming, online exams, and other scenarios.

Most users do not have the conditions to set up a green screen. Through the arbitrary background segmentation capability provided by ZEGO, they can identify the subject in the image without a green screen.

Suitable for online education, video conferencing, and other scenarios.

Illustration

Feature Scenarios

Based on the object segmentation capability, developers can implement business scenarios such as background blurring, virtual background, presenter mode, and multi-person real-time co-stage interaction, creating more diverse interactive experiences.

Feature PointBackground BlurVirtual BackgroundBackground TransparencySubject Segmentation and Transmission
Feature DescriptionBlur the image outside the subject.Replace the image outside the subject with custom images, videos, or colors.

Render the subject's image on other video content locally.

For example, implement presenter mode and other functions on screen sharing or playing video content.

Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented subject in the image to the play stream end, and render the subject at the play stream end, achieving the visual effect of multiple people in different locations being in the same scene in real time
Illustration

Hardware Compatibility

PlatformHardware Requirements
Android

  • Snapdragon chips:

    • Snapdragon 6 series: Snapdragon 675 and above
    • Snapdragon 7 series: Snapdragon 730 and above
    • Snapdragon 8 series: Snapdragon 835 and above

  • HiSilicon Kirin chips:

    • Kirin 8 series: Kirin 820 and above
    • Kirin 9 series: Kirin 980 and above

  • MediaTek chips:

    • Helio P series: Helio P60 and above
    • Dimensity series: Dimensity 820 and above
  • Samsung chips: Exynos 1080 and above
iOSA series chips: Apple A9 and above, for example iPhone 6s
macOSM series chips: Apple M1 and above
WindowsIntel Core i5 and above

Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Others/src/main/java/com/example/others/videoobjectsegmentation" directory.

Prerequisites

Before using the object segmentation feature, please ensure:

Implementation Flow

Warning
  • Enabling the object segmentation feature will consume additional system resources. To ensure user experience, currently only supports enabling object segmentation for one channel's published stream image.
  • For Android 12 and above versions, you need to add the following configuration in AndroidManifest.xml. For details, please refer to Android Developers related instructions。
    <uses-native-library android:name="libOpenCL.so" android:required="false" />
    <uses-native-library android:name="libOpenCL-pixel.so" android:required="false" />
  • If there are third-party filters that have undergone custom preprocessing, you need to ensure that the third-party filters support Alpha channel pass-through functionality.

Please note that developers can choose whether to implement the (Optional) steps in the above figure according to their business scenario needs. If implementation is needed, please refer to the specific instructions below.

1 Initialize and login to Room

For the specific process of initialization and logging in to Room, please refer to "Create Engine" and "Login to Room" in the implementing video call documentation.

2 Make special settings for view

Before enabling object segmentation, you need to set the view for rendering in advance. Currently, both TextureView and SurfaceView are supported.

  • If the view is of TextureView type, you need to call setOpaque to set the opaque attribute to false.
TextureView playerView;//View for playing
playerView.setOpaque(false);
  • If the view is of SurfaceView type, you need to call setFormat to set the PixelFormat attribute to TRANSLUCENT, and you need to place the view at the topmost layer of the display window.
SurfaceView playerView;//View for playing
playerView.getHolder().setFormat(PixelFormat.TRANSLUCENT);
playerView.setZOrderOnTop(true)//Place SurfaceView at the topmost layer of the display window

3 (Optional) Implement custom rendering class

Note

Through the following method, the performance of custom rendering is relatively low. ZEGO recommends that you use OpenGL ES for rendering to improve rendering performance. For details, please refer to Run Example Source Code implementation.

Developers can perform custom rendering according to actual conditions. It should be noted that the video raw data contains an Alpha channel, and the raw data has not undergone Alpha premultiplication processing. Taking drawing Bitmap to render play stream data as an example:

  1. Listen to custom rendering data callback and cache rendering frame data.

    Warning

    Do not render directly in the callback, as it will block the callback thread and cause rendering abnormalities.

    @Override
public void onRemoteVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZegoVideoFrameParam param, String streamID) {
    super.onRemoteVideoFrameRawData(data, dataLength, param, streamID);

    if(param.format == ZegoVideoFrameFormat.RGBA32)
    {
        RenderConfig renderConfig = new RenderConfig();
        renderConfig.frameParam = param;
        renderConfig.buffer = new byte[data[0].capacity()];
        data[0].get(renderConfig.buffer,0,data[0].capacity());
        // Cache playStream1 play stream data
        if(streamID.equals(playStream1)){
            dataQueuePlay1.offer(renderConfig);
        }
        // Cache playStream2 play stream data
        if(streamID.equals(playStream2)){
            dataQueuePlay2.offer(renderConfig);
        }
    }
}

  2. Process the original video frame data byte by byte and convert it to ARGB_8888 type data in 4-byte units.

    int[] array = new int[renderFrame.buffer.length/4];

for(int i=0;i<renderFrame.buffer.length/4;i++){
    b =  renderFrame.buffer[i*4];
    g =  renderFrame.buffer[i*4+1];
    r =  renderFrame.buffer[i*4+2];
    a =  renderFrame.buffer[i*4+3];

    int dd = (a&0xff)<<24 | (b&0xff)<<16 | (g&0xff)<<8 | (r&0xff);
    array[i] = (dd);
}

Bitmap bmp = Bitmap.createBitmap(array, renderFrame.frameParam.width, renderFrame.frameParam.height, Bitmap.Config.ARGB_8888);

  3. Periodically draw the bitmap to the rendering view.

    Canvas canvas = playView1.lockCanvas();
if(canvas != null){
    // Clear screen
    paintPlayView1.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.CLEAR));
    canvas.drawPaint(paintPlayView1);

    //Off-screen drawing
    int layerID = canvas.saveLayer(0,0,canvas.getWidth(),canvas.getHeight(), null,Canvas.ALL_SAVE_FLAG);
    //Select SRC mode
    paintPlayView1.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.SRC));
    canvas.drawBitmap(bmpFinal, 0,0, paintPlayView1);
    paintPlayView1.setXfermode(null);
    canvas.restoreToCount(layerID);
}
playView1.unlockCanvasAndPost(canvas);

4 (Optional) Enable custom rendering

Call the enableCustomVideoRender interface to set advanced engine configuration and enable custom rendering, and call the setCustomVideoRenderHandler interface to set custom video rendering callbacks. For details, please refer to Custom Video Rendering.

Warning

When using custom rendering for object segmentation, bufferType only supports the ZegoVideoBufferType.RAW_DATA type.

videoRenderer = new CustomVideoRender();//Custom video rendering class
videoRenderer.setPreviewView(previewView);//Preview view
videoRenderer.setPlayView1(playView1);//Play stream view1
videoRenderer.setPlayView2(playView2);//Play stream view2
ZegoCustomVideoRenderConfig config = new ZegoCustomVideoRenderConfig();//Custom rendering configuration
config.bufferType = ZegoVideoBufferType.RAW_DATA;//Select RAW_DATA type video frame data
config.frameFormatSeries = ZegoVideoFrameFormatSeries.RGB;//Select RGB color series data format
engine.enableCustomVideoRender(true, config);//Start custom video rendering
engine.setCustomVideoRenderHandler(videoRenderer);//Set custom video rendering callback

5 Listen to object segmentation state callback

Call the onVideoObjectSegmentationStateChanged interface to listen to object segmentation state callbacks.

Warning

The object segmentation state callback depends on enabling preview or publishing stream. That is, if you need to listen to the onVideoObjectSegmentationStateChanged callback, you need to call preview startPreview or publishing stream startPublishingStream.

@Override
public void onVideoObjectSegmentationStateChanged(ZegoObjectSegmentationState state, ZegoPublishChannel channel, int errorCode) {
    super.onVideoObjectSegmentationStateChanged(state, channel, errorCode);

    if(state == ZegoObjectSegmentationState.ON){
        //Object segmentation enabled
    }else{
        //Object segmentation disabled
        //Abnormal shutdown, please check the error code
    }
}

6 Use object segmentation to implement different business functions

Warning

If developers need to update the object segmentation type or background processing type, they need to modify the configuration of ZegoObjectSegmentationConfig and call the enableVideoObjectSegmentation interface again to enable object segmentation to update the object segmentation effect; the update result will be notified to developers through the onVideoObjectSegmentationStateChanged callback.

Background Blur

Call the enableVideoObjectSegmentation interface to enable object segmentation and set the background processing type to "blur".

ZegoObjectSegmentationConfig config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.ANY_BACKGROUND;//Select the object segmentation type to enable according to the actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.BLUR;//Set background processing mode to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevel.MEDIUM;//Set background blur level to medium
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.MAIN);//Enable object segmentation

Virtual Background

Virtual background supports two types of materials:

  • Images. Currently supports "PNG" and "JPEG" image formats, that is, image files with ".png", ".jpg", and ".jpeg" extensions.
  • Videos, with the following restrictions:
    • Video format: MP4, FLV, MKV, AVI.
    • Video source: Local video.
    • Video playback mode: Loop playback.
    • Resolution: Maximum not exceeding 4096 px, recommended within 1920 px.
    • Video duration: Maximum not exceeding 30 seconds, recommended within 15 seconds.
    • Video size: Maximum not exceeding 50 MB, recommended within 10 MB.
Warning

When developers use this feature, please pay attention to the aspect ratio of custom images and video materials, otherwise parts exceeding the view will be cropped.

Call the enableVideoObjectSegmentation interface to enable object segmentation and set the background processing type to "image" or "video".

ZegoObjectSegmentationConfig config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.ANY_BACKGROUND;//Select the object segmentation type to enable according to the actual situation

//Set background processing mode to image
config.backgroundConfig.processType = ZegoBackgroundProcessType.IMAGE;
config.backgroundConfig.imageURL = "<image_path>";//Set background image path
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.MAIN);//Enable object segmentation

//Set background processing mode to video
config.backgroundConfig.processType = ZegoBackgroundProcessType.VIDEO;
config.backgroundConfig.videoURL = "<video_path>";//Set background video path
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.MAIN);//Enable object segmentation

Transparent Background

Warning

If developers need to implement business functions similar to "presenter mode", they need to mix the "subject image" and "video source content to be mixed" into one video stream on the business side.

Call the enableVideoObjectSegmentation interface to enable object segmentation and set the background processing type to "transparent".

ZegoObjectSegmentationConfig config = new ZegoObjectSegmentationConfig();
config.objectSegmentationType = ZegoObjectSegmentationType.ANY_BACKGROUND;//Select the object segmentation type to enable according to the actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessType.TRANSPARENT;//Set background processing mode to transparent
engine.enableVideoObjectSegmentation(enable, config, ZegoPublishChannel.MAIN);//Enable object segmentation

7 (Optional) Use Alpha channel to transmit segmented subject

If the publishing end needs to transmit the segmented subject image to the play stream end through the Alpha channel and render the subject at the play stream end, you need to first call the enableAlphaChannelVideoEncoder interface to set the encoder to support transparent channel, then call the startPublishingStream interface to publish stream, so that it can be smoothly transmitted to the play stream end.

Warning

Currently only supports transparent channel data arranged below RGB or YUV data.

  • Enable Alpha channel data transmission:

    ZegoAlphaLayoutType layoutType = ZegoAlphaLayoutType.BOTTOM; // Transparent channel data arranged below RGB or YUV data
engine.enableAlphaChannelVideoEncoder(true, layoutType, ZegoPublishChannel.MAIN); // Enable encoder to support transparent channel

  • Disable Alpha channel data transmission:

    ZegoAlphaLayoutType layoutType = ZegoAlphaLayoutType.BOTTOM; // Transparent channel data arranged below RGB or YUV data
engine.enableAlphaChannelVideoEncoder(false, layoutType, ZegoPublishChannel.MAIN); // Disable encoder to support transparent channel

8 Start preview and publishing stream

After enabling the object segmentation feature through the enableVideoObjectSegmentation interface, you can preview.

Note

Developers can also enable preview first, then enable object segmentation. This article takes enabling object segmentation first, then previewing as an example for introduction.

  • Method 1: Use internal rendering

    If using internal rendering, please set alphaBlend to true before starting preview.

    ZegoCanvas canvas = new ZegoCanvas(previewView);
canvas.alphaBlend = true;//Enable internal rendering Alpha blending. After enabling, it supports Alpha blending between the segmented subject and the background layer
engine.startPreview(canvas);
engine.startPublishingStream(streamID);

  • Method 2: Use custom rendering

    If using custom rendering, you can directly preview and publish stream.

    engine.startPreview();
engine.startPublishingStream(streamID);

9 (Optional) Set Alpha channel rendering at play stream end and play stream

Warning

Only when the publishing end has enabled Alpha channel transmission is it necessary to enable Alpha channel rendering at the play stream end.

  • Method 1: Use internal rendering

    If using internal rendering, before calling the startPlayingStream interface to start playing stream at the play stream end, you need to set the alphaBlend attribute of ZegoCanvas to true.

    ZegoCanvas canvas = new ZegoCanvas(playView);
canvas.alphaBlend = true;//Enable internal rendering Alpha blending. After enabling, it supports Alpha blending between the segmented subject and the background layer
engine.startPlayingStream(streamID, canvas);

  • Method 2: Use custom rendering

    If using custom rendering, you can directly play stream.

    engine.startPlayingStream(streamID, null);

10 Disable object segmentation

Call the enableVideoObjectSegmentation interface to disable object segmentation.

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationType.ANY_BACKGROUND;//Select the object segmentation type to disable according to the actual situation
engine.enableVideoObjectSegmentation(false, objectType, ZegoPublishChannel.MAIN);//Disable object segmentation

11 Destroy Engine

Call the destroyEngine interface to destroy the Engine.

ZegoExpressEngine.destroyEngine(null);

Previous

Super Resolution

Next

H.265

On this page

Back to top