Publish Stream Video Enhancement
Feature Overview
ZEGO Express SDK provides multiple video preprocessing enhancement capabilities. Developers can adjust the image effects at the publishing end according to business needs.
| Feature | Feature Description | Feature Advantages | Effect Display | |
|---|---|---|---|---|
| Basic Beauty Effects | According to user and business needs, adjust the degree of whitening, smoothing, sharpening and rosiness, easily achieve basic beauty effects, present good skin condition to users, and create unique and natural beauty effects. | Cover high-frequency beauty capabilities. |  | |
| Low Light Enhancement | When ambient light is relatively dark and the brightness of the image captured by the camera does not meet business needs such as seeing faces clearly or performing face recognition, enhance the image brightness. |
|  |  |
| Video Noise Reduction | Poor camera capture effects, low ambient light brightness, etc., may cause obvious noise in the image. The video noise reduction feature can reduce image noise. |
|  |  |
| Color Enhancement | Due to camera characteristics, captured video may have insufficient saturation. Using color enhancement feature, while protecting skin tone, enhance undersaturated colors to make image colors more realistic and more in line with human visual perception. |
|  |  |
Prerequisites
Before using the publish stream video enhancement feature, please ensure:
- You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
- You have integrated ZEGO Express SDK in your project and implemented basic audio and video streaming functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
Usage Steps
Initialization and Login to Room
For the specific process of initialization and logging in to Room, please refer to "Create Engine" and "Login to Room" in the "Implementing Video Call" document.
Set Publish Stream Video Enhancement Effects
Please choose the corresponding feature to configure according to your needs.
This feature is the basic beauty feature of ZEGO Express SDK built-in ZEGO Effects SDK, only including four beauty effects: whitening, rosiness, smoothing, and sharpening. If you need to use more advanced beauty features, please refer to Using Real-Time Video and AI Beauty Effects Together.
Initialize Basic Beauty Environment
If you need to use the basic beauty feature, you must call the startEffectsEnv interface to initialize the beauty environment before startPreview starts preview and startPublishingStream publishes stream.
// Create beauty environment
ZegoExpressEngine.getEngine().startEffectsEnv();After initializing the beauty environment, the SDK will use a fixed video frame data type for transmission. If you need to use video custom preprocessing when using the basic beauty feature, you can only configure the corresponding video frame data type:
Android platform: Only supports Texture 2D type data, that is, the bufferType of ZegoCustomVideoCaptureConfig is set to "GL_TEXTURE_2D".
Enable/Disable Basic Beauty Effects
The calling order of the enableEffectsBeauty interface and setEffectsBeautyParam interface has no priority.
After initializing the beauty environment, you can call the enableEffectsBeauty interface before or after publishing stream to enable or disable beauty effects in real time.
// Enable/disable beauty effects
ZegoExpressEngine.getEngine().enableEffectsBeauty(true);Set Basic Beauty Effect Parameters
After initializing the beauty environment, you can call the setEffectsBeautyParam interface before or after publishing stream to set beauty effect parameters in real time.
- smoothIntensity: Smoothing, perform smoothing while retaining facial details, such as retaining moles on the face.
- whitenIntensity: Whitening, whiten the face by increasing the overall brightness of the image.
- rosyIntensity: Rosiness, perform warm color processing on the overall image.
- sharpenIntensity: Sharpening, perform sharpening processing on the overall image. When the image is somewhat blurry, you can slightly increase sharpening to make the outline clear.
The value range of the above four parameters is 0 to 100. The larger the value, the higher the degree of beauty. The default value is 50.
// Create beauty parameter object
ZegoEffectsBeautyParam beautyParam = new ZegoEffectsBeautyParam();
// Whitening, rosiness, smoothing, sharpening
beautyParam.whitenIntensity = 50;
beautyParam.rosyIntensity = 50;
beautyParam.smoothIntensity = 50;
beautyParam.sharpenIntensity = 50;
// Set beauty parameters
ZegoExpressEngine.getEngine().setEffectsBeautyParam(beautyParam);(Optional) Destroy Basic Beauty Environment
When the enableEffectsBeauty interface is called and set to "false", the beauty effect will be turned off, but the beauty module will still occupy resources and consume performance. If you want to completely release resources and save performance consumption, you need to call the stopEffectsEnv interface to destroy the beauty environment before preview and publishing stream.
When destroyEngine is called to destroy the engine, the SDK will automatically destroy the beauty environment.
// Destroy beauty environment
ZegoExpressEngine.getEngine().stopEffectsEnv();ZEGO provides two ways of low light enhancement:
| Method | Description |
|---|---|
| Enable adaptive frame rate for enhancement | Dynamically reduce capture frame rate within the set frame rate range to extend the exposure time of each frame, achieving the effect of enhancing the brightness of the captured image.
For example: The developer enables and sets the adaptive frame rate range to [10, 25]. When ambient light is sufficient, the capture frame rate is still 25 frames. When ambient light brightness is low, the capture frame rate will be dynamically adjusted between [10, 25] to extend the exposure time of each frame to enhance the brightness of the captured image. |
| Algorithm enhances captured image brightness |
|
Both processing methods can be used independently or simultaneously, and both are compatible with beauty features.
Set Low Light Enhancement Mode
RTC SDK supports different algorithm low light enhancement modes. Call the setLowlightEnhancement interface to set the low light enhancement mode ZegoLowlightEnhancementMode:
- Before using the setLowlightEnhancement interface, ensure that the SDK includes video internal capture and video internal filter modules.
- If the business uses externally captured video streams, only specific data types
PIXEL_BUFFER_TYPE_GL_TEXTURE_2DandPIXEL_BUFFER_TYPE_MEMare supported.
| Enumeration Value | Description |
|---|---|
| ZegoLowlightEnhancementMode.OFF | Turn off low light enhancement feature. |
| ZegoLowlightEnhancementMode.ON | Enable low light enhancement feature, controlled by developers. |
| ZegoLowlightEnhancementMode.AUTO | Smart control of low light enhancement feature, intelligently enable or disable the feature based on image lighting conditions. |
Low light enhancement type ZegoExpLowlightEnhancementType includes the following types:
| Enumeration Value | Description |
|---|---|
| ZegoExpLowlightEnhancementType.Normal | Regular low light enhancement. |
| ZegoExpLowlightEnhancementType.AI | AI low light enhancement. Note If you need to use this feature, please contact ZEGO Technical Support. |
Taking enabling automatic low light enhancement feature as an example:
ZegoExpLowlightEnhancementParams params = new ZegoExpLowlightEnhancementParams();
params.mode = ZegoLowlightEnhancementMode.AUTO;
params.type = ZegoExpLowlightEnhancementType.AI;
engine.setLowlightEnhancementParams(params, ZegoPublishChannel.MAIN);(Optional) Enable Camera Adaptive Frame Rate
When ambient light brightness is low and the camera capture frame rate is too high, the exposure time of each frame is insufficient, which may cause the captured video image to be dark. At this time, you can enable camera adaptive frame rate through enableCameraAdaptiveFPS. The SDK will match the camera-supported capture frame rate range according to the set frame rate range. Within this range, dynamically adjust the camera capture frame rate according to ambient brightness to improve image brightness when the set frame rate is too high.
//Set minimum frame rate
int minCamFPS = getIntent().getIntExtra("camMinFPS", 0);
//Set maximum frame rate
int maxCamFPS = getIntent().getIntExtra("camMaxFPS", 0);
engine.enableCameraAdaptiveFPS(true, minCamFPS, maxCamFPS, ZegoPublishChannel.MAIN);Call the setVideoDenoiseParams interface to set video noise reduction parameters ZegoVideoDenoiseParams including: video noise reduction mode ZegoVideoDenoiseMode and video noise reduction intensity ZegoVideoDenoiseStrength.
- Before using the setVideoDenoiseParams interface, ensure that the SDK includes video internal capture and video internal filter modules.
- If the business uses externally captured video streams, only specific data types
PIXEL_BUFFER_TYPE_GL_TEXTURE_2DandPIXEL_BUFFER_TYPE_MEMare supported. - Video noise reduction feature is not enabled by default, that is, the default value of ZegoVideoDenoiseMode is
ZegoVideoDenoiseMode.off.
Taking setting automatic noise reduction as an example:
ZegoVideoDenoiseParams p = new ZegoVideoDenoiseParams();
p.mode = ZegoVideoDenoiseMode.AUTO;
p.strength = ZegoVideoDenoiseStrength.LIGHT;
engine.setVideoDenoiseParams(p, ZegoPublishChannel.MAIN);Call the enableColorEnhancement interface to set color enhancement parameters ZegoColorEnhancementParams (parameter properties are as shown in the following table) and other parameters to enable the color enhancement feature. Color enhancement feature is not enabled by default.
| Color Enhancement Parameter | Description |
|---|---|
| intensity | Intensity of color enhancement, value range [0.0, 1.0], default is 0.0. The larger the value, the greater the color enhancement effect. |
| skinToneProtectionLevel | Intensity of skin tone protection, value range [0.0, 1.0], default is 1.0. The larger the value, the better the protection effect. |
| lipColorProtectionLevel | Intensity of lip color protection, value range [0.0, 1.0], default is 0.0. The larger the value, the better the protection effect. |
ZegoColorEnhancementParams p = new ZegoColorEnhancementParams();
p.intensity = 1;//Value range: [0,1], the larger the value, the greater the color enhancement intensity. Default value: 0.
p.skinToneProtectionLevel = 1;//Value range: [0,1], the larger the value, the greater the skin tone protection degree. Default value: 1.
p.lipColorProtectionLevel = 1;//Value range: [0,1], the larger the value, the greater the lip color protection degree. Default value: 0.
engine.enableColorEnhancement(true, p, ZegoPublishChannel.MAIN);//Enable color enhancementStart Preview
After calling the startPreview interface to start preview, you can set different video enhancement features and experience the effects in real time.
ZegoCanvas previewCanvas = new ZegoCanvas(preview);
engine.startPreview(previewCanvas);Start Publishing Stream
When the preview effect meets expectations, you can call the startPublishingStream interface to start publishing stream. The publishing stream image effect will be consistent with the preview effect.
engine.startPublishingStream("STREAM_ID");The above implements the video enhancement feature before publishing stream.