Publishing Stream Video Enhancement

Initialize basic AI effects environment

If you need to use the basic AI effects feature, you must call the startEffectsEnv interface before startPreview starts preview and startPublishingStream starts publishing stream to initialize the AI effects environment.

// Create AI effects environment
[[ZegoExpressEngine sharedInstance] startEffectsEnv];

// Create AI effects environment
[[ZegoExpressEngine sharedInstance] startEffectsEnv];

After initializing the AI effects environment, the SDK internally uses a fixed video frame data type for transmission. If you need to use custom video preprocessing while using the basic AI effects feature, you can only configure the corresponding video frame data type:

iOS and macOS platforms: Only support CVPixelBuffer type data, that is, the bufferType of ZegoCustomVideoCaptureConfig takes the value "ZegoVideoBufferTypeCVPixelBuffer".

Enable basic AI effects

After initializing the AI effects environment, you can call the enableEffectsBeauty interface before or after publishing stream to enable or disable AI effects in real time.

// Toggle AI effects
[[ZegoExpressEngine sharedInstance] enableEffectsBeauty:true];

// Toggle AI effects
[[ZegoExpressEngine sharedInstance] enableEffectsBeauty:true];

Set basic AI effects parameters

After initializing the AI effects environment, you can call the setEffectsBeautyParam interface before or after publishing stream to set AI effects parameters in real time.

• smoothIntensity: Smoothing, perform smoothing while retaining facial details, such as retaining moles on the face.
• whitenIntensity: Whitening, increase the overall brightness of the picture to whiten the face.
• rosyIntensity: Rosyness, perform warm color processing on the overall picture.
• sharpenIntensity: Sharpening, perform sharpening processing on the overall picture. When the picture is slightly blurry, you can slightly increase sharpening to make the outline clear.

The value range of the above four parameters is 0 to 100. The larger the value, the higher the degree of AI effects. The default value is 50.

// Create AI effects parameter object
ZegoEffectsBeautyParam *beautyParam = [[ZegoEffectsBeautyParam alloc] init];

// Whitening, rosyness, smoothing, sharpening
beautyParam.whitenIntensity = 50;
beautyParam.rosyIntensity = 50;
beautyParam.smoothIntensity = 50;
beautyParam.sharpenIntensity = 50;

// Set AI effects parameters
[[ZegoExpressEngine sharedInstance] setEffectsBeautyParam:beautyParam];

// Create AI effects parameter object
ZegoEffectsBeautyParam *beautyParam = [[ZegoEffectsBeautyParam alloc] init];

// Whitening, rosyness, smoothing, sharpening
beautyParam.whitenIntensity = 50;
beautyParam.rosyIntensity = 50;
beautyParam.smoothIntensity = 50;
beautyParam.sharpenIntensity = 50;

// Set AI effects parameters
[[ZegoExpressEngine sharedInstance] setEffectsBeautyParam:beautyParam];

(Optional) Destroy basic AI effects environment

When calling the enableEffectsBeauty interface and setting it to "false", the AI effects will be turned off, but the AI effects module will still occupy resources and consume performance. If you want to completely release resources and save performance consumption, you need to call the stopEffectsEnv interface before preview and publishing stream to destroy the AI effects environment.

// Destroy AI effects environment
[[ZegoExpressEngine sharedInstance] stopEffectsEnv];

// Destroy AI effects environment
[[ZegoExpressEngine sharedInstance] stopEffectsEnv];

ZEGO provides two ways of low light enhancement:

The two processing methods can be used independently or simultaneously, and both are compatible with AI effects features.

Set low light enhancement switch

RTC SDK supports different algorithm low light enhancement modes. Call the setLowlightEnhancement interface to set the low light enhancement mode ZegoLowlightEnhancementMode:

The low light enhancement type ZegoExpLowlightEnhancementType includes the following types:

Taking enabling automatic low light enhancement feature as an example:

ZegoExpLowlightEnhancementParams *params = [[ZegoExpLowlightEnhancementParams alloc] init];
params.mode = ZegoLowlightEnhancementModeAuto;
params.type = ZegoExpLowlightEnhancementTypeAI;
[[ZegoExpressEngine sharedEngine] setLowlightEnhancementParams:params channel:ZegoPublishChannelMain];

ZegoExpLowlightEnhancementParams *params = [[ZegoExpLowlightEnhancementParams alloc] init];
params.mode = ZegoLowlightEnhancementModeAuto;
params.type = ZegoExpLowlightEnhancementTypeAI;
[[ZegoExpressEngine sharedEngine] setLowlightEnhancementParams:params channel:ZegoPublishChannelMain];

(Optional) Enable camera adaptive frame rate

When the ambient light brightness is low and the camera capture frame rate is too high, the exposure time of each frame is insufficient, which may cause the captured video picture to be dark. At this time, you can enable camera adaptive frame rate through enableCameraAdaptiveFPS. The SDK will match the capture frame rate range supported by the camera according to the set frame rate range. Within this range, dynamically adjust the camera capture frame rate according to the ambient brightness to improve the picture brightness when the set frame rate is too high.

//Set minimum frame rate
int minCamFPS = 15;
//Set maximum frame rate
int maxCamFPS = 35;
[[ZegoExpressEngine sharedEngine] enableCameraAdaptiveFPS:YES minFPS:minCamFPS maxFPS:maxCamFPS channel:ZegoPublishChannelMain];

//Set minimum frame rate
int minCamFPS = 15;
//Set maximum frame rate
int maxCamFPS = 35;
[[ZegoExpressEngine sharedEngine] enableCameraAdaptiveFPS:YES minFPS:minCamFPS maxFPS:maxCamFPS channel:ZegoPublishChannelMain];

Call the setVideoDenoiseParams interface to set video noise reduction parameters ZegoVideoDenoiseParams, including: video noise reduction mode ZegoVideoDenoiseMode and video noise reduction strength ZegoVideoDenoiseStrength.

Taking automatic noise reduction as an example:

ZegoVideoDenoiseParams *p = [[ZegoVideoDenoiseParams alloc] init];
p.mode = ZegoVideoDenoiseModeAuto;
p.strength = ZegoVideoDenoiseStrengthLight;
[[ZegoExpressEngine sharedEngine] setVideoDenoiseParams:p channel:ZegoPublishChannelMain];

ZegoVideoDenoiseParams *p = [[ZegoVideoDenoiseParams alloc] init];
p.mode = ZegoVideoDenoiseModeAuto;
p.strength = ZegoVideoDenoiseStrengthLight;
[[ZegoExpressEngine sharedEngine] setVideoDenoiseParams:p channel:ZegoPublishChannelMain];

Call the enableColorEnhancement interface, set color enhancement parameters ZegoColorEnhancementParams (parameter properties are as shown in the table below), and other parameters to enable the color enhancement feature. Color enhancement feature is not enabled by default.

ZegoColorEnhancementParams *p = [[ZegoColorEnhancementParams alloc]init];
p.intensity = 1f;//Value range: [0,1], the larger the value, the greater the color enhancement intensity. Default value: 0.
p.skinToneProtectionLevel = 1f;//Value range: [0,1], the larger the value, the greater the skin tone protection degree. Default value: 1.
p.lipColorProtectionLevel = 1f;//Value range: [0,1], the larger the value, the greater the lip color protection degree. Default value: 0.
[[ZegoExpressEngine sharedEngine]enableColorEnhancement:YES params:p channel:ZegoPublishChannelMain];//Enable color enhancement

ZegoColorEnhancementParams *p = [[ZegoColorEnhancementParams alloc]init];
p.intensity = 1f;//Value range: [0,1], the larger the value, the greater the color enhancement intensity. Default value: 0.
p.skinToneProtectionLevel = 1f;//Value range: [0,1], the larger the value, the greater the skin tone protection degree. Default value: 1.
p.lipColorProtectionLevel = 1f;//Value range: [0,1], the larger the value, the greater the lip color protection degree. Default value: 0.
[[ZegoExpressEngine sharedEngine]enableColorEnhancement:YES params:p channel:ZegoPublishChannelMain];//Enable color enhancement