Subject Segmentation

Feature Introduction

Subject segmentation is a value-added capability provided by Express SDK. It uses AI algorithms to recognize content in video frames and sets transparency information for each pixel. Pixels in the subject portion are set to "opaque", while pixels outside the subject portion are set to "transparent". Developers can use the transparency information of these pixels to process the subject and non-subject parts differently, thereby implementing various functions.

Subject Segmentation Objects

For users in different environments, ZEGOCLOUD provides two segmentation capabilities: "green screen background segmentation" and "arbitrary background segmentation".

Segmentation Type	Green Screen Background Segmentation	Arbitrary Background Segmentation
Capability Description	When users have set up a green screen themselves, the subject in non-green screen areas can be retained. Suitable for e-commerce live streaming, online exams, and other scenarios.	Most users do not have the conditions to set up a green screen. Through ZEGOCLOUD's arbitrary background segmentation capability, the subject in the frame can be recognized without a green screen. Suitable for online education, video conferences, and other scenarios.
Illustration

Feature Scenarios

Based on subject segmentation capability, developers can implement business scenarios such as background blurring, virtual background, presenter mode, and multi-person real-time co-location interaction, creating more diverse interactive experiences.

Feature Point	Background Blur	Virtual Background	Background Transparency	Subject Segmentation and Transmission
Feature Description	Blur the area outside the subject.	Replace the area outside the subject with custom images, videos, or colors.	Render the subject's image on top of other video content locally. For example, implement presenter mode and other functions on top of screen sharing or playing video content.	Combine with the Alpha channel data transmission capability provided by Express SDK to transmit the segmented subject to the playing end for subject rendering, achieving the visual effect of multiple people in different locations appearing in the same scene in real time
Illustration

Hardware Compatibility

Sample Source Code

Please refer to Download Sample Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Examples/Others/VideoObjectSegmentation" directory.

Prerequisites

Before using the subject segmentation feature, please ensure:

• You have contacted ZEGOCLOUD Technical Support for special packaging.

• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.

• You have integrated ZEGO Express SDK in the project and implemented basic audio and video stream publishing and playing functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation Flow.

Implementation Flow

Please note that developers can choose whether to implement the (Optional) steps in the above diagram based on their business scenario needs. If implementation is needed, please refer to the specific instructions below.

1 Initialize and login room

For the specific process of initialization and login room, please refer to "Create engine" and "Login room" in the video call documentation.

2 Make special settings for view

Before enabling subject segmentation, you need to set the view used for rendering, i.e., call setBackgroundColor to set the backgroundColor property to clearColor.

@property (weak, nonatomic) IBOutlet UIView *view;
[self.view setBackgroundColor:UIColor.clearColor];

@property (weak, nonatomic) IBOutlet UIView *view;
[self.view setBackgroundColor:UIColor.clearColor];

3 (Optional) Implement custom rendering class

- (void)onCapturedVideoFrameRawData:(unsigned char * _Nonnull *)data dataLength:(unsigned int *)dataLength param:(ZegoVideoFrameParam *)param flipMode:(ZegoVideoFlipMode)flipMode channel:(ZegoPublishChannel)channel {
    if (param.format == ZegoVideoFrameFormatBGRA32) {
        CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
        CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, data[0], dataLength[0], NULL);

        CGImageRef cgImageFromBytes = CGImageCreate((int)param.size.width, (int)param.size.height, 8, 32, param.strides[0], colorSpace, kCGImageByteOrder32Little|kCGImageAlphaFirst, dataProvider, NULL, NO, kCGRenderingIntentDefault);

        UIImage *image = [UIImage imageWithCGImage:cgImageFromBytes];

        // Draw image to rendering view
        dispatch_async(dispatch_get_main_queue(), ^{
            self.previewView.image = image;
            CGImageRelease(cgImageFromBytes);
        });
    }
}

- (void)onCapturedVideoFrameRawData:(unsigned char * _Nonnull *)data dataLength:(unsigned int *)dataLength param:(ZegoVideoFrameParam *)param flipMode:(ZegoVideoFlipMode)flipMode channel:(ZegoPublishChannel)channel {
    if (param.format == ZegoVideoFrameFormatBGRA32) {
        CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
        CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, data[0], dataLength[0], NULL);

        CGImageRef cgImageFromBytes = CGImageCreate((int)param.size.width, (int)param.size.height, 8, 32, param.strides[0], colorSpace, kCGImageByteOrder32Little|kCGImageAlphaFirst, dataProvider, NULL, NO, kCGRenderingIntentDefault);

        UIImage *image = [UIImage imageWithCGImage:cgImageFromBytes];

        // Draw image to rendering view
        dispatch_async(dispatch_get_main_queue(), ^{
            self.previewView.image = image;
            CGImageRelease(cgImageFromBytes);
        });
    }
}

4 (Optional) Enable custom rendering

ZegoCustomVideoRenderConfig *renderConfig = [[ZegoCustomVideoRenderConfig alloc] init];
renderConfig.bufferType = ZegoVideoBufferTypeRawData;// Select RAW_DATA type video frame data
renderConfig.frameFormatSeries = ZegoVideoFrameFormatSeriesRGB;// Select RGB color series data format
[[ZegoExpressEngine sharedEngine] enableCustomVideoRender:YES config:renderConfig];// Start custom video rendering
[[ZegoExpressEngine sharedEngine] setCustomVideoRenderHandler:self];// Set custom video rendering callback

ZegoCustomVideoRenderConfig *renderConfig = [[ZegoCustomVideoRenderConfig alloc] init];
renderConfig.bufferType = ZegoVideoBufferTypeRawData;// Select RAW_DATA type video frame data
renderConfig.frameFormatSeries = ZegoVideoFrameFormatSeriesRGB;// Select RGB color series data format
[[ZegoExpressEngine sharedEngine] enableCustomVideoRender:YES config:renderConfig];// Start custom video rendering
[[ZegoExpressEngine sharedEngine] setCustomVideoRenderHandler:self];// Set custom video rendering callback

5 Listen for subject segmentation state callback

Call the onVideoObjectSegmentationStateChanged interface to listen for subject segmentation state callbacks.

-(void)onVideoObjectSegmentationStateChanged:(ZegoObjectSegmentationState)state channel:(ZegoPublishChannel)channel errorCode:(int)errorCode{
    if(state == ZegoObjectSegmentationStateOn){
        //Subject segmentation enabled
    }else{
        //Subject segmentation disabled
        //Abnormal shutdown, please check the error code
    }
}

-(void)onVideoObjectSegmentationStateChanged:(ZegoObjectSegmentationState)state channel:(ZegoPublishChannel)channel errorCode:(int)errorCode{
    if(state == ZegoObjectSegmentationStateOn){
        //Subject segmentation enabled
    }else{
        //Subject segmentation disabled
        //Abnormal shutdown, please check the error code
    }
}

6 Use subject segmentation to implement different business functions

Background Blur

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeBlur;// Set background processing method to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevelMedium;// Set background blur level to medium
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeBlur;// Set background processing method to blur
config.backgroundConfig.blurLevel = ZegoBackgroundBlurLevelMedium;// Set background blur level to medium
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

Virtual Background

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to enable based on actual situation

//Set background processing method to image
config.backgroundConfig.processType = ZegoBackgroundProcessTypeImage;
config.backgroundConfig.imageUrl= @"<image_path>";// Set background image path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

//Set background processing method to video
config.backgroundConfig.processType = ZegoBackgroundProcessTypeVideo;
config.backgroundConfig.videoURL= @"<video_path>";// Set background video path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to enable based on actual situation

//Set background processing method to image
config.backgroundConfig.processType = ZegoBackgroundProcessTypeImage;
config.backgroundConfig.imageUrl= @"<image_path>";// Set background image path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

//Set background processing method to video
config.backgroundConfig.processType = ZegoBackgroundProcessTypeVideo;
config.backgroundConfig.videoURL= @"<video_path>";// Set background video path
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

Transparent Background

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeTransparent;// Set background processing method to transparent
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

ZegoObjectSegmentationConfig *config = [[ZegoObjectSegmentationConfig alloc]init];
config.objectSegmentationType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to enable based on actual situation
config.backgroundConfig.processType = ZegoBackgroundProcessTypeTransparent;// Set background processing method to transparent
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:YES config:config channel:ZegoPublishChannelMain];// Enable subject segmentation

7 (Optional) Use Alpha channel to transmit the segmented subject

8 Start preview and publish stream

After enabling the subject segmentation feature through the enableVideoObjectSegmentation interface, you can preview.

• Method 1: Use internal rendering

  If using internal rendering, before enabling preview, please set alphaBlend to YES.

  ZegoCanvas *previewCanvas = [ZegoCanvas canvasWithView:self.previewView];
  previewCanvas.alphaBlend = YES;// Enable internal rendering Alpha blending, after enabling supports Alpha blending of the segmented subject with background layer
  [[ZegoExpressEngine sharedEngine]startPreview:previewCanvas];
  [[ZegoExpressEngine sharedEngine] startPublishingStream:streamID];

  ZegoCanvas *previewCanvas = [ZegoCanvas canvasWithView:self.previewView];
  previewCanvas.alphaBlend = YES;// Enable internal rendering Alpha blending, after enabling supports Alpha blending of the segmented subject with background layer
  [[ZegoExpressEngine sharedEngine]startPreview:previewCanvas];
  [[ZegoExpressEngine sharedEngine] startPublishingStream:streamID];

• Method 2: Use custom rendering

  If using custom rendering, you can directly preview and publish stream.

  [[ZegoExpressEngine sharedEngine] startPreview:nil];
  [[ZegoExpressEngine sharedEngine] startPublishingStream:streamID];

  [[ZegoExpressEngine sharedEngine] startPreview:nil];
  [[ZegoExpressEngine sharedEngine] startPublishingStream:streamID];

9 (Optional) Set Alpha channel rendering at the playing end and play stream

10 Disable subject segmentation

Call the enableVideoObjectSegmentation interface to disable subject segmentation.

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to disable based on actual situation
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:NO type:objectType channel:ZegoPublishChannelMain];// Disable subject segmentation

ZegoObjectSegmentationType objectType = ZegoObjectSegmentationTypeAnyBackground;// Select the subject segmentation type to disable based on actual situation
[[ZegoExpressEngine sharedEngine]enableVideoObjectSegmentation:NO type:objectType channel:ZegoPublishChannelMain];// Disable subject segmentation

11 Destroy engine

Call the destroyEngine interface to destroy the engine.

[ZegoExpressEngine destroyEngine:nil];

[ZegoExpressEngine destroyEngine:nil];