Background segmentation

The ZegoEffects SDK provides various background segmentation features that can be used in a wide range of use cases, such as online education, video conferencing, short videos, and more.

Prerequisites

Before you begin to use the SDK's background segmentation feature in your project, make sure you complete the following steps:

• Integrate the ZegoEffects SDK into your project. For details, see Integration.
• Implement the basic image processing functionality. For details, see Implementation.
• Import the SegmentationModel. For details, see Import resources and models.

Implementation steps

The SDK supports two background segmentation methods: portrait segmentation and green screen segmentation. For each segmentation method, you can also enable the background blurring or background mosaicing feature as needed.

Portrait segmentation

1. Do either one of the following:

   • Call the setPortraitSegmentationBackgroundPath method to set the background path,
   • Call the setPortraitSegmentationBackgroundBuffer method to set the background buffer.
   Untitled

   // Method 1: Set the background path for portrait segmentation. 
   [self.effects setPortraitSegmentationBackgroundPath: @"MY_BACKGROUND_PATH" mode:ZegoEffectsScaleModeAspectFill];
   
   // Method 2: Set the background buffer for portrait segmentation. 
   [self.effects setPortraitSegmentationBackgroundBuffer:pixelbuffer mode:ZegoEffectsScaleModeAspectFill];
   

   1

   Copied!
   Untitled

   // Method 1: Set the background path for portrait segmentation. 
   mEffects.setPortraitSegmentationBackgroundPath("MY_BACKGROUND_PATH", ZegoEffectsScaleMode.ASPECT_FILL);
   
   // Method 2: Set the background buffer for portrait segmentation.
   mEffects.setPortraitSegmentationBackgroundBuffer(buffer, bufferLength, param, ZegoEffectsScaleMode.ASPECT_FILL);
   
   // Method 3: Set the background texture for portrait segmentation. 
   mEffects.setPortraitSegmentationBackgroundTexture(textureID, param, ZegoEffectsScaleMode.ASPECT_FILL);
   

   1

   Copied!

2. Call the enablePortraitSegmentation method to enable the portrait segmentation feature.

   Untitled

   // Enable the portrait segmentation feature.
   [self.effects enablePortraitSegmentation:YES];
   

   1

   Copied!
   Untitled

   // Turn on the portrait split function
   mEffects.enablePortraitSegmentation(true);
   

   1

   Copied!

After enabling portrait segmentation, you can also enable background blurring or background mosaicing as needed.

Background blurring with portrait segmentation

1. Call the enablePortraitSegmentationBackgroundBlur method to enable background blurring with portrait segmentation.
2. Call the setPortraitSegmentationBackgroundBlurParam method to set the intensity property of ZegoEffectsBlurParam to adjust the extent of background blurring.

// Enable background blurring with portrait segmentation.
[self.effects enablePortraitSegmentationBackgroundBlur:YES];

// Set the background blurring parameters.
ZegoEffectsBlurParam *param = [[ZegoEffectsBlurParam alloc] init];
param.intensity = 100;
[self.effects setPortraitSegmentationBackgroundBlurParam:param];

// Enable background blurring with portrait segmentation.
mEffects.enablePortraitSegmentationBackgroundBlur(true);

// Set the background blurring parameters.
ZegoEffectsBlurParam param = new ZegoEffectsBlurParam();
param.intensity = 100;
mEffects.setPortraitSegmentationBackgroundBlurParam(param);

Background mosaicing with portrait segmentation

1. Call the enablePortraitSegmentationBackgroundMosaic method to enable background mosaicing with portrait segmentation.
2. Call the setPortraitSegmentationBackgroundMosaicParam method to set the intensity property of ZegoEffectsMosaicParam to adjust the extent of background mosaicing.

// Enable background mosaicing with portrait segmentation.
[self.effects enablePortraitSegmentationBackgroundMosaic:YES];
  
// Set the background mosaicing parameters.
ZegoEffectsMosaicParam *param = [[ZegoEffectsMosaicParam alloc] init];
param.intensity = 100;
[self.effects setPortraitSegmentationBackgroundMosaicParam:param];

// Enable background mosaicing with portrait segmentation.
mEffects.enablePortraitSegmentationBackgroundMosaic(true);

// Set the background mosaicing parameters.
ZegoEffectsMosaicParam param = new ZegoEffectsMosaicParam();
param.intensity = 100;
mEffects.setPortraitSegmentationBackgroundMosaicParam(param);

Green screen segmentation

1. Do either one of the following:

   • Call the setChromaKeyBackgroundPath method to set the background path.
   • Call the setChromaKeyBackgroundBuffer method to set the background background buffer.
   • Call the setChromaKeyBackgroundPath method to set the background path.
   • Call the setChromaKeyBackgroundBuffer method to set the background background buffer.
   • Call the setChromaKeyBackgroundTexture method to set the background texture.
   Untitled

   // Method 1: Set the background path for green screen segmentation.
   [self.effects setChromaKeyBackgroundPath: @"MY_BACKGROUND_PATH" mode:ZegoEffectsScaleModeAspectFill];
   
   // Method 2: Set the background buffer for green screen segmentation.
   [self.effects setChromaKeyBackgroundBuffer:pixelbuffer mode:ZegoEffectsScaleModeAspectFill];
   

   1

   Copied!
   Untitled

   // Method 1: Set the background path for the green screen segmentation.
   mEffects.setChromaKeyBackgroundPath("MY_BACKGROUND_PATH", ZegoEffectsScaleMode.ASPECT_FILL);
   
   // Method 2: Set the background buffer for green scceen segmentation.
   mEffects.setChromaKeyBackgroundBuffer(buffer, bufferLength, param, ZegoEffectsScaleMode.ASPECT_FILL);
   
   // Method 3: Set the background Texture for green screen segmentation.
   mEffects.setChromaKeyBackgroundTexture(textureID, param, ZegoEffectsScaleMode.ASPECT_FILL);
   

   1

   Copied!

2. Call the enableChromaKey method to enable the green screen segmentation feature, and then call the setChromaKeyParam method to set the chrome key parameters to adjust the color tolerance, edge smoothness index, and others.

   Untitled

   // Enable the green screen segmentation feature.
   [self.effects enableChromaKey:YES]
   
   // Set the green screen segmentation parameters.
   ZegoEffectsChromaKeyParam *param = [[ZegoEffectsChromaKeyParam alloc] init];
   [self.effects setChromaKeyParam:param];
   

   1

   Copied!
   Untitled

   // Enable the green screen segmentation feature.
   mEffects.enableChromaKey(true);
   
   // Set the green screen segmentation parameters.
   ZegoEffectsChromaKeyParam param = new ZegoEffectsChromaKeyParam();
   mEffects.setChromaKeyParam(param);
   

   1

   Copied!

After enabling green screen segmentation, you can also enable background blurring or background mosaicing as needed.

Background blurring with green screen segmentation

1. Call the enableChromaKeyBackgroundBlur method to enable background blurring with green screen segmentation.
2. Call the setChromaKeyBackgroundBlurParam method to set the intensity property of ZegoEffectsBlurParam to adjust the extent of background blurring.

// Enable background blurring with green screen segmentation.
[self.effects enableChromaKeyBackgroundBlur:YES];
    
// Set the background blurring parameters.
ZegoEffectsBlurParam *param = [[ZegoEffectsBlurParam alloc] init];
param.intensity = 100;
[self.effects setChromaKeyBackgroundBlurParam:param];

// Enable background blurring with green screen segmentation.
mEffects.enableChromaKeyBackgroundBlur(true);

// Set the background blurring parameters.
ZegoEffectsBlurParam param = new ZegoEffectsBlurParam();
param.intensity = 100;
mEffects.setChromaKeyBackgroundBlurParam(param);

Background mosaicing with green screen segmentation

1. Call the enableChromaKeyBackgroundMosaic method to enable the background mosaicing feature.
2. Call the setChromaKeyBackgroundMosaicParam method to set the intensity property of ZegoEffectsMosaicParam to adjust the extent of background mosaicing.

// Enable background mosaicing with green screen segmentation.
[self.effects enableChromaKeyBackgroundMosaic:YES];
// Set the background mosaicing parameters.
ZegoEffectsMosaicParam *param = [[ZegoEffectsMosaicParam alloc] init];
param.intensity = 100;
[self.effects setChromaKeyBackgroundMosaicParam:param];

// Enable background mosaicing with green screen segmentation.
mEffects.enableChromaKeyBackgroundMosaic(true);

// Set the background mosaicing parameters.
ZegoEffectsMosaicParam param = new ZegoEffectsMosaicParam();
param.intensity = 100;
mEffects.setChromaKeyBackgroundMosaicParam(param);