Subject Segmentation

2024-09-06

Introduction

Subject Segmentation is a value-added capability provided by Express SDK. It uses AI algorithms to identify content in video frames and sets transparency information for each pixel. Pixels of the main subject are set to "opaque", while pixels outside the subject are set to "transparent". Developers can use this transparency information to process the subject and non-subject parts differently, thereby implementing various features.

Warning

The current official SDK does not include "Subject Segmentation" related features. If needed, please contact ZEGO Technical Support for special packaging and provide your AppID to enable relevant permissions.
The "Subject Segmentation" feature is a paid feature. For trial applications or formal pricing inquiries, please contact ZEGO sales personnel.

Subject Segmentation Types

For users in different environments, ZEGO provides two segmentation capabilities: "Green Screen Background Segmentation" and "Arbitrary Background Segmentation".

Segmentation Type	Green Screen Background Segmentation	Arbitrary Background Segmentation
Capability Description	When users set up a green screen themselves, the subject in non-green screen areas can be retained. Suitable for e-commerce live streaming, online exams, and other scenarios.	Most users don't have the conditions to set up a green screen. Through ZEGO's arbitrary background segmentation capability, they can identify subjects in the frame without a green screen. Suitable for online education, video conferences, and other scenarios.
Illustration

Segmentation Type

Green Screen Background Segmentation

Arbitrary Background Segmentation

Capability Description

When users set up a green screen themselves, the subject in non-green screen areas can be retained.

Suitable for e-commerce live streaming, online exams, and other scenarios.

Most users don't have the conditions to set up a green screen. Through ZEGO's arbitrary background segmentation capability, they can identify subjects in the frame without a green screen.

Suitable for online education, video conferences, and other scenarios.

Illustration

Feature Scenarios

Based on the subject segmentation capability, developers can implement business scenarios such as background blurring, virtual background, presenter mode, and multi-person real-time co-hosting, creating more diverse interactive experiences.

Feature	Background Blur	Virtual Background	Transparent Background	Subject Segmentation and Transmission
Feature Description	Apply blur processing to areas outside the subject.	Replace areas outside the subject with custom images, videos, or colors.	Render the subject's visuals on top of other video content locally. For example, implement features like presenter mode on top of screen sharing or playing video content.	Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented subject to the playing end, where subject rendering is performed, achieving the visual effect of multiple people in different locations being in the same scene in real time
Illustration

Feature

Background Blur

Virtual Background

Transparent Background

Subject Segmentation and Transmission

Feature Description

Apply blur processing to areas outside the subject.

Replace areas outside the subject with custom images, videos, or colors.

Render the subject's visuals on top of other video content locally.

For example, implement features like presenter mode on top of screen sharing or playing video content.

Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented subject to the playing end, where subject rendering is performed, achieving the visual effect of multiple people in different locations being in the same scene in real time

Illustration

Hardware Compatibility

Platform	Hardware Requirements
Android	Qualcomm chips: Snapdragon 6 series: Snapdragon 675 and above Snapdragon 7 series: Snapdragon 730 and above Snapdragon 8 series: Snapdragon 835 and above HiSilicon Kirin chips: Kirin 8 series: Kirin 820 and above Kirin 9 series: Kirin 980 and above MediaTek chips: Helio P series: Helio P60 and above Dimensity series: Dimensity 820 and above Samsung chips: Exynos 1080 and above
iOS	A series chips: Apple A9 and above, for example iPhone 6s
macOS	M series chips: Apple M1 and above
Windows	Intel Core i5 and above

Sample Source Code

Please refer to Download Sample Source Code to obtain the source code.

For related source code, please check files in the "/ZegoExpressExample/Others/src/main/java/com/example/others/videoobjectsegmentation" directory.

Prerequisites

Before using the subject segmentation feature, please ensure:

You have contacted ZEGO Technical Support for special packaging.
The processor is M1 or above.
A project has been created in the ZEGO Console, and valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.
ZEGO Express SDK has been integrated into the project, and basic audio/video streaming functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation Flow.

Implementation Flow

Warning

Enabling subject segmentation will consume additional system resources. To ensure user experience, currently only supports enabling subject segmentation for publishing stream on one channel.
If using third-party filters with custom pre-processing, ensure the third-party filters support Alpha channel pass-through functionality.

Please note that developers can choose whether to implement the (Optional) steps in the diagram above according to their business scenario needs. If implementation is needed, please refer to the specific instructions below.

For the specific process of initialization and room login, please refer to "Create Engine" and "Login Room" in the implementing video call documentation.

(Optional) Implement Custom Rendering Class

Implement Custom Rendering Class

Developers can perform custom rendering according to actual situations. Note that the raw video data contains an Alpha channel, and the raw data has not undergone Alpha premultiplication processing. Taking rendering playing stream data by drawing NSImage as an example:

Listen for custom rendering data callbacks.

Warning

Do not render directly in the callback, as it will block the callback thread and cause rendering abnormalities.

- (void)onRemoteVideoFrameRawData:(unsigned char * _Nonnull *)data dataLength:(unsigned int *)dataLength param:(ZegoVideoFrameParam *)param streamID:(NSString *)streamID
{

}

- (void)onRemoteVideoFrameRawData:(unsigned char * _Nonnull *)data dataLength:(unsigned int *)dataLength param:(ZegoVideoFrameParam *)param streamID:(NSString *)streamID
{

}

Cache the raw video frame data.

auto video_frame = std::make_shared<ZegoVideoFrameBuffer>();
video_frame->data = std::make_unique<uint8_t[]>(dataLength[0]);
video_frame->data_length =dataLength[0];
video_frame->format = param.format;
video_frame->width = (int)param.size.width;
video_frame->height = (int)param.size.height;
video_frame->bitsPerPixel = param.strides[0];

memcpy(video_frame->data.get(), data[0], dataLength[0]);
{
    // Cache one frame
    std::lock_guard<std::mutex> lock(play_video_frame_mutex_);
    play_video_frames_[[streamID UTF8String]] = video_frame;
}

auto video_frame = std::make_shared<ZegoVideoFrameBuffer>();
video_frame->data = std::make_unique<uint8_t[]>(dataLength[0]);
video_frame->data_length =dataLength[0];
video_frame->format = param.format;
video_frame->width = (int)param.size.width;
video_frame->height = (int)param.size.height;
video_frame->bitsPerPixel = param.strides[0];

memcpy(video_frame->data.get(), data[0], dataLength[0]);
{
    // Cache one frame
    std::lock_guard<std::mutex> lock(play_video_frame_mutex_);
    play_video_frames_[[streamID UTF8String]] = video_frame;
}

Periodically draw the bitmap to the rendering view.

dispatch_async(dispatch_get_main_queue(), ^{
    std::shared_ptr<ZegoVideoFrameBuffer> frame;
    {
       std::lock_guard<std::mutex> lock(self->play_video_frame_mutex_);
        frame = self->play_video_frames_[[streamID UTF8String]];
        if (!frame) {
            return;
        }
        self->play_video_frames_[[streamID UTF8String]].reset();
    }

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, frame->data.get(), frame->data_length, NULL);
    CGImageRef cgImageFromBytes = CGImageCreate(frame->width, frame->height, 8, 32, frame->bitsPerPixel, colorSpace, kCGImageByteOrder32Little|kCGImageAlphaFirst, dataProvider, NULL, NO, kCGRenderingIntentDefault);

    NSImage *finalImage = [[NSImage alloc]initWithCGImage:cgImageFromBytes size:NSMakeSize(frame->width, frame->height)];
    CGDataProviderRelease(dataProvider);
    CGImageRelease(cgImageFromBytes);

    self.playView1.layer.contentsGravity = kCAGravityResizeAspect;
    self.playView1.layer.contents = finalImage;
});

dispatch_async(dispatch_get_main_queue(), ^{
    std::shared_ptr<ZegoVideoFrameBuffer> frame;
    {
       std::lock_guard<std::mutex> lock(self->play_video_frame_mutex_);
        frame = self->play_video_frames_[[streamID UTF8String]];
        if (!frame) {
            return;
        }
        self->play_video_frames_[[streamID UTF8String]].reset();
    }

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, frame->data.get(), frame->data_length, NULL);
    CGImageRef cgImageFromBytes = CGImageCreate(frame->width, frame->height, 8, 32, frame->bitsPerPixel, colorSpace, kCGImageByteOrder32Little|kCGImageAlphaFirst, dataProvider, NULL, NO, kCGRenderingIntentDefault);

    NSImage *finalImage = [[NSImage alloc]initWithCGImage:cgImageFromBytes size:NSMakeSize(frame->width, frame->height)];
    CGDataProviderRelease(dataProvider);
    CGImageRelease(cgImageFromBytes);

    self.playView1.layer.contentsGravity = kCAGravityResizeAspect;
    self.playView1.layer.contents = finalImage;
});

(Optional) Enable Custom Rendering

Enable Custom Rendering

Call the enableCustomVideoRender interface, set engine advanced configuration and enable custom rendering, and call the setCustomVideoRenderHandler interface to set custom video rendering callbacks. For details, please refer to Custom Video Rendering.

Warning

When using custom rendering for subject segmentation, bufferType only supports the ZegoVideoBufferType.RAW_DATA type.

ZegoCustomVideoRenderConfig *renderConfig = [[ZegoCustomVideoRenderConfig alloc] init];
renderConfig.bufferType = ZegoVideoBufferTypeRawData;// Select RAW_DATA type video frame data
renderConfig.frameFormatSeries = ZegoVideoFrameFormatSeriesRGB;// Select RGB color series data format
[[ZegoExpressEngine sharedEngine] enableCustomVideoRender:YES config:renderConfig];// Start custom video rendering
[[ZegoExpressEngine sharedEngine] setCustomVideoRenderHandler:self];// Set custom video rendering callback

ZegoCustomVideoRenderConfig *renderConfig = [[ZegoCustomVideoRenderConfig alloc] init];
renderConfig.bufferType = ZegoVideoBufferTypeRawData;// Select RAW_DATA type video frame data
renderConfig.frameFormatSeries = ZegoVideoFrameFormatSeriesRGB;// Select RGB color series data format
[[ZegoExpressEngine sharedEngine] enableCustomVideoRender:YES config:renderConfig];// Start custom video rendering
[[ZegoExpressEngine sharedEngine] setCustomVideoRenderHandler:self];// Set custom video rendering callback

Listen for Subject Segmentation State Callback

Call the onVideoObjectSegmentationStateChanged interface to listen for subject segmentation state callbacks.

Warning

The subject segmentation state callback depends on enabling preview or publishing stream. That is, to listen for the onVideoObjectSegmentationStateChanged callback, you need to call preview startPreview or publish stream startPublishingStream.

-(void)onVideoObjectSegmentationStateChanged:(ZegoObjectSegmentationState)state channel:(ZegoPublishChannel)channel errorCode:(int)errorCode{
    if(state == ZegoObjectSegmentationStateOn){
        //Subject segmentation enabled
    }else{
        //Subject segmentation disabled
        //Abnormal shutdown, please check the error code
    }
}

-(void)onVideoObjectSegmentationStateChanged:(ZegoObjectSegmentationState)state channel:(ZegoPublishChannel)channel errorCode:(int)errorCode{
    if(state == ZegoObjectSegmentationStateOn){
        //Subject segmentation enabled
    }else{
        //Subject segmentation disabled
        //Abnormal shutdown, please check the error code
    }
}

Use Subject Segmentation to Implement Different Business Features

Warning

If developers need to update the subject segmentation type or background processing type, they need to modify the configuration of ZegoObjectSegmentationConfig and call the enableVideoObjectSegmentation interface again to enable subject segmentation, which can update the subject segmentation effect; the update result will be notified to developers through the onVideoObjectSegmentationStateChanged callback.

Background Blur

Use Subject Segmentation to Implement Background Blur

Call the enableVideoObjectSegmentation interface to enable subject segmentation and set the background processing type to "Blur".

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to enable according to actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeBlur;//Set background processing method to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevelMedium;//Set background blur level to medium
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to enable according to actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeBlur;//Set background processing method to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevelMedium;//Set background blur level to medium
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

Virtual Background

Use Subject Segmentation to Implement Virtual Background

Virtual background supports two types of materials:

Images. Currently supports "PNG" and "JPEG" image formats, that is, image files with ".png", ".jpg", and ".jpeg" extensions.
Videos, with the following restrictions:
- Video format: MP4, FLV, MKV, AVI.
- Video source: Local video.
- Video playback method: Loop playback.
- Resolution: Maximum not exceeding 4096 px, recommended within 1920 px.
- Video duration: Maximum not exceeding 30 seconds, recommended within 15 seconds.
- Video size: Maximum not exceeding 50 MB, recommended within 10 MB.

Warning

When using this feature, developers should pay attention to the aspect ratio of custom image and video materials, otherwise parts exceeding the view will be cropped.

Call the enableVideoObjectSegmentation interface to enable subject segmentation and set the background processing type to "Image" or "Video".

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to enable according to actual situation

//Set background processing method to Image
config.backgroundConfig.processType = ZegoBackgroundProcessTypeImage;
config.backgroundConfig.imageUrl= @"<image_path>";//Set background image path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

//Set background processing method to Video
config.backgroundConfig.processType = ZegoBackgroundProcessTypeVideo;
config.backgroundConfig.videoURL= @"<video_path>";//Set background video path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to enable according to actual situation

//Set background processing method to Image
config.backgroundConfig.processType = ZegoBackgroundProcessTypeImage;
config.backgroundConfig.imageUrl= @"<image_path>";//Set background image path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

//Set background processing method to Video
config.backgroundConfig.processType = ZegoBackgroundProcessTypeVideo;
config.backgroundConfig.videoURL= @"<video_path>";//Set background video path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

Transparent Background

Use Subject Segmentation to Implement Transparent Background

Warning

If developers need to implement business features similar to "presenter mode", they need to mix the "subject visuals" and "video source content to be mixed" into one video stream on the business side.

Call the enableVideoObjectSegmentation interface to enable subject segmentation and set the background processing type to "Transparent".

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to enable according to actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeTransparent;//Set background processing method to transparent
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to enable according to actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeTransparent;//Set background processing method to transparent
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];//Enable subject segmentation

(Optional) Use Alpha Channel to Transmit Segmented Subject

Use Alpha Channel to Transmit Segmented Subject

If the publishing end needs to transmit the segmented subject visuals to the playing end through the Alpha channel for subject rendering at the playing end, you need to first call the enableAlphaChannelVideoEncoder interface to set the encoder to support transparent channel, then call the startPublishingStream interface to publish stream, enabling smooth transmission to the playing end.

Warning

Currently only supports transparent channel data arranged below RGB or YUV data.

Enable Alpha channel data transmission:

ZegoAlphaLayoutType layoutType = ZegoAlphaLayoutTypeBottom; // Transparent channel data arranged below RGB or YUV data
[[ZegoExpressEngine sharedEngine]enableAlphaChannelVideoEncoder:YES alphaLayout:layoutType channel:ZegoPublishChannelMain]; // Set encoder to support transparent channel

ZegoAlphaLayoutType layoutType = ZegoAlphaLayoutTypeBottom; // Transparent channel data arranged below RGB or YUV data
[[ZegoExpressEngine sharedEngine]enableAlphaChannelVideoEncoder:YES alphaLayout:layoutType channel:ZegoPublishChannelMain]; // Set encoder to support transparent channel

Disable Alpha channel data transmission:

ZegoAlphaLayoutType layoutType = ZegoAlphaLayoutTypeBottom; // Transparent channel data arranged below RGB or YUV data
[[ZegoExpressEngine sharedEngine]enableAlphaChannelVideoEncoder:NO alphaLayout:layoutType channel:ZegoPublishChannelMain]; // Set encoder to support transparent channel

ZegoAlphaLayoutType layoutType = ZegoAlphaLayoutTypeBottom; // Transparent channel data arranged below RGB or YUV data
[[ZegoExpressEngine sharedEngine]enableAlphaChannelVideoEncoder:NO alphaLayout:layoutType channel:ZegoPublishChannelMain]; // Set encoder to support transparent channel

Start Preview and Publish Stream

After enabling the subject segmentation feature through the enableVideoObjectSegmentation interface, you can perform preview.

Note

Developers can also enable preview first, then enable subject segmentation. This article uses enabling subject segmentation first, then preview as an example.

Method 1: Use internal rendering

If using internal rendering, before enabling preview, please set alphaBlend to YES.

ZegoCanvas *previewCanvas = [ZegoCanvas canvasWithView:self.previewView];
previewCanvas.alphaBlend = YES;//Enable internal rendering Alpha blending. After enabling, supports Alpha blending of segmented subject with background layer
[[ZegoExpressEngine sharedEngine]startPreview:previewCanvas];

ZegoCanvas *previewCanvas = [ZegoCanvas canvasWithView:self.previewView];
previewCanvas.alphaBlend = YES;//Enable internal rendering Alpha blending. After enabling, supports Alpha blending of segmented subject with background layer
[[ZegoExpressEngine sharedEngine]startPreview:previewCanvas];

Method 2: Use custom rendering

If using custom rendering, you can directly perform preview and publish stream.

[[ZegoExpressEngine sharedEngine] startPreview:nil];
[[ZegoExpressEngine sharedEngine] startPublishingStream:streamID];

[[ZegoExpressEngine sharedEngine] startPreview:nil];
[[ZegoExpressEngine sharedEngine] startPublishingStream:streamID];

(Optional) Set Alpha Channel Rendering at Playing End and Start Playing Stream

Set Alpha Channel Rendering at Playing End and Start Playing Stream

Warning

Alpha channel rendering at the playing end only needs to be enabled when the publishing end has enabled Alpha channel transmission.

Method 1: Use internal rendering

If using internal rendering, before the playing end calls the startPlayingStream interface to start playing stream, it needs to set the alphaBlend property of ZegoCanvas to YES.

ZegoCanvas *playCanvas = [ZegoCanvas canvasWithView:self.playView1];
playCanvas.alphaBlend = YES;//Enable internal rendering Alpha blending. After enabling, supports Alpha blending of segmented subject with background layer
[[ZegoExpressEngine sharedEngine]startPlayingStream:streamID  canvas:playCanvas];

ZegoCanvas *playCanvas = [ZegoCanvas canvasWithView:self.playView1];
playCanvas.alphaBlend = YES;//Enable internal rendering Alpha blending. After enabling, supports Alpha blending of segmented subject with background layer
[[ZegoExpressEngine sharedEngine]startPlayingStream:streamID  canvas:playCanvas];

Method 2: Use custom rendering

If using custom rendering, you can directly start playing stream.

[[ZegoExpressEngine sharedEngine]startPlayingStream:streamID canvas:nil];

[[ZegoExpressEngine sharedEngine]startPlayingStream:streamID canvas:nil];

Disable Subject Segmentation

Call the enableVideoObjectSegmentation interface to disable subject segmentation.

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to disable according to actual situation
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:NO type:objectType channel:ZegoPublishChannelMain];//Disable subject segmentation

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationTypeAnyBackground;//Select the subject segmentation type to disable according to actual situation
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:NO type:objectType channel:ZegoPublishChannelMain];//Disable subject segmentation

Destroy Engine

Call the destroyEngine interface to destroy the engine.

[ZegoExpressEngine destroyEngine:nil];

[ZegoExpressEngine destroyEngine:nil];

Subject Segmentation

Introduction

Subject Segmentation Types

Feature Scenarios

Hardware Compatibility

Sample Source Code

Prerequisites

Implementation Flow

Initialize and Login Room

(Optional) Implement Custom Rendering Class

(Optional) Enable Custom Rendering

Listen for Subject Segmentation State Callback

Use Subject Segmentation to Implement Different Business Features

Background Blur

Virtual Background

Transparent Background

(Optional) Use Alpha Channel to Transmit Segmented Subject

Start Preview and Publish Stream

(Optional) Set Alpha Channel Rendering at Playing End and Start Playing Stream

Disable Subject Segmentation

Destroy Engine