Publishing Stream Video Enhancement
Feature Overview
ZEGO Express SDK provides multiple video pre-processing enhancement capabilities. Developers can adjust the screen effects on the publishing side according to business needs.
| Feature | Feature Description | Feature Advantages | Effect Demo | |
|---|---|---|---|---|
| Basic AI Effects | According to user and business needs, adjust the degree of whitening, smoothing, sharpening, and rosyness, easily achieve basic AI effects functionality, present users with good skin condition, and create unique natural AI effects. | Covers high-frequency AI effects capabilities. |  | |
| Low Light Enhancement | When ambient light is dark and the brightness of the screen captured by the camera does not meet the needs of seeing faces clearly or performing face recognition and other business needs, enhance the screen brightness. |
|  |  |
| Color Enhancement | Due to camera characteristics, captured video may have insufficient saturation. Use the color enhancement feature to enhance undersaturated colors while protecting skin tone, making screen colors more realistic and more in line with human visual perception. |
|  |  |
Prerequisites
Before using the publishing stream video enhancement feature, ensure that:
- A project has been created in the ZEGOCLOUD Console, and a valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.
- ZEGO Express SDK has been integrated into the project, and basic audio/video stream publishing and playing functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
Implementation Steps
Initialize and login to room
For the specific process of initialization and logging in to the room, please refer to "Create Engine" and "Login Room" in the implementing video call documentation.
Set publishing stream video enhancement effects
Please select the corresponding feature to configure according to your needs.
This feature is the basic AI effects feature of ZEGO Effects SDK built into ZEGO Express SDK, including only four AI effects: whitening, rosyness, smoothing, and sharpening. For more advanced AI effects features, please refer to Using Real-Time Audio/Video with AI Effects.
Initialize basic AI effects environment
If you need to use the basic AI effects feature, you must call the startEffectsEnv interface to initialize the AI effects environment before starting preview with startPreview and publishing stream with startPublishingStream.
// Create AI effects environment
engine->startEffectsEnv();After initializing the AI effects environment, the SDK will use a fixed video frame data type for transmission internally. If you need to use video custom pre-processing while using the basic AI effects feature, you can only configure the corresponding video frame data type:
Windows platform: Only supports video frame raw data, i.e., the bufferType value of ZegoCustomVideoCaptureConfig is "ZEGO_VIDEO_BUFFER_TYPE_RAW_DATA".
Enable/disable basic AI effects
There is no order requirement for calling the enableEffectsBeauty interface and the setEffectsBeautyParam interface.
After initializing the AI effects environment, you can call the enableEffectsBeauty interface to enable or disable AI effects in real-time before or after publishing stream.
// Enable/disable AI effects
engine->enableEffectsBeauty(true);Set basic AI effects parameters
After initializing the AI effects environment, you can call the setEffectsBeautyParam interface to set AI effects parameters in real-time before or after publishing stream.
- smoothIntensity: Smoothing, perform smoothing while retaining facial details, such as retaining moles on the face.
- whitenIntensity: Whitening, whiten the face by increasing the overall brightness of the screen.
- rosyIntensity: Rosyness, apply warm color processing to the overall screen.
- sharpenIntensity: Sharpening, apply sharpening to the overall screen. When the screen is slightly blurry, you can increase sharpening slightly to make contours clear.
The value range of all four parameters above is 0 ~ 100. The larger the value, the higher the degree of AI effects. The default value is 50.
// Create AI effects parameter object
ZegoEffectsBeautyParam beautyParam;
// Whitening, rosyness, smoothing, sharpening
beautyParam.whitenIntensity = 50;
beautyParam.rosyIntensity = 50;
beautyParam.smoothIntensity = 50;
beautyParam.sharpenIntensity = 50;
// Set AI effects parameters
engine->setEffectsBeautyParam(beautyParam);（Optional）Destroy basic AI effects environment
When the enableEffectsBeauty interface is called and set to "false", the AI effects will be turned off, but the AI effects module will still occupy resources and consume performance. If you want to completely release resources and save performance consumption, you need to call the stopEffectsEnv interface to destroy the AI effects environment before preview and publishing stream.
When calling destroyEngine to destroy the engine, the SDK will automatically destroy the AI effects environment.
// Destroy AI effects environment
engine->stopEffectsEnv();ZEGO provides two methods of low light enhancement:
| Method | Description |
|---|---|
| Enable adaptive frame rate for enhancement | Dynamically reduce the capture frame rate within the set frame rate range to extend the exposure time of each frame, achieving the effect of enhancing the brightness of the captured screen.
For example: The developer enables and sets the adaptive frame rate range to [10, 25]. When ambient light is sufficient, the capture frame rate is still 25 frames. When ambient light is low, the capture frame rate will be dynamically adjusted between [10, 25] to extend the exposure time of each frame to enhance the brightness of the captured screen. |
| Algorithm enhances captured screen brightness |
|
Both processing methods can be used independently or simultaneously, and both are compatible with AI effects functionality.
Set low light enhancement mode
RTC SDK supports different algorithm low light enhancement modes. Call the setLowlightEnhancement interface to set the low light enhancement mode ZegoLowlightEnhancementMode:
- Before using the setLowlightEnhancement interface, ensure that the SDK includes the video internal capture and video internal filter modules.
- If the business uses external capture for video streams, only the specific data type
PIXEL_BUFFER_TYPE_MEMis supported.
| Enumeration Value | Description |
|---|---|
| ZEGO_LOWLIGHT_ENHANCEMENT_MODE_OFF | Disable low light enhancement feature, capture screen brightness remains unchanged. |
| ZEGO_LOWLIGHT_ENHANCEMENT_MODE_ON | Enable low light enhancement feature, capture screen brightness is enhanced. |
| ZEGO_LOWLIGHT_ENHANCEMENT_MODE_AUTO | Automatic low light enhancement feature, capture screen brightness enhancement value dynamically adjusts with ambient brightness. |
Taking enabling automatic low light enhancement feature as an example:
engine->setLowlightEnhancement(ZEGO_LOWLIGHT_ENHANCEMENT_MODE_AUTO);
params.mode = ZEGO_LOWLIGHT_ENHANCEMENT_MODE_AUTO;
params.type = ZEGO_EXP_LOWLIGHT_ENHANCEMENT_TYPE_AI
engine->setLowlightEnhancementParams(params);【New: Select low light algorithm example code】
（Optional）Enable camera adaptive frame rate
When ambient light is low and the camera capture frame rate is too high, the exposure time of each frame is insufficient, which may cause the captured video screen to be dark. At this time, you can enable camera adaptive frame rate through enableCameraAdaptiveFPS. The SDK will match the camera's supported capture frame rate range according to the set frame rate range. Within this range, dynamically adjust the camera capture frame rate according to ambient brightness to improve screen brightness when the set frame rate is too high.
int minCamFPS = 15;//Set minimum frame rate
int maxCamFPS = 35;//Set maximum frame rate
engine->enableCameraAdaptiveFPS(true, minCamFPS, maxCamFPS);Call the enableColorEnhancement interface, set the color enhancement parameters ZegoColorEnhancementParams (parameter attributes are as follows in the table), and other parameters to enable the color enhancement feature. Color enhancement feature is not enabled by default.
| Color Enhancement Parameter | Description |
|---|---|
| intensity | Intensity of color enhancement, value range [0.0, 1.0], default is 0.0. The larger the value, the greater the color enhancement effect. |
| skinToneProtectionLevel | Intensity of skin tone protection, value range [0.0, 1.0], default is 1.0. The larger the value, the better the protection effect. |
| lipColorProtectionLevel | Intensity of lip color protection, value range [0.0, 1.0], default is 0.0. The larger the value, the better the protection effect. |
ZegoColorEnhancementParams p;
p.intensity = 1;//Value range: [0,1], the larger the value, the greater the color enhancement intensity. Default value: 0.
p.skinToneProtectionLevel = 1;//Value range: [0,1], the larger the value, the greater the skin tone protection. Default value: 1.
p.lipColorProtectionLevel = 1;//Value range: [0,1], the larger the value, the greater the lip color protection. Default value: 0.
engine->enableColorEnhancement(true, p, ZEGO_PUBLISH_CHANNEL_MAIN);//Enable color enhancementStart preview
Call the startPreview interface to enable preview. You can set different video enhancement features and experience the effects in real-time.
ZegoCanvas previewCanvas(preview);
engine->startPreview(&previewCanvas);Start publishing stream
When the preview effect meets expectations, you can call the startPublishingStream interface to start publishing stream. The publishing stream screen effect will be consistent with the preview effect.
engine->startPublishingStream("STREAM_ID");The above implements the video enhancement feature before publishing stream.