Super Resolution

Feature Introduction

Super Resolution (SR) function can multiply the width and height pixels of the video stream picture at the playing stream end. For example: If the original picture resolution pulled by the playing stream end is 640p x 360p, after super resolution processing, the resolution will be increased to 1280p x 720p.

Effect Display

• Portrait

Before Super Resolution	After Super Resolution

• Text

Before Super Resolution	After Super Resolution

Application Scenarios

• 1v1 video call scenarios
• Live streaming scenarios: In most cases, only 1 stream is played in live streaming scenarios. The super resolution feature can be enabled for the single stream live picture. When playing multiple streams, the super resolution feature can be enabled for the focused streamer.
• Online education: Online education scenarios may have multiple streams, but there will be 1 focus stream (such as a teacher). You can choose to enable the super resolution feature for the teacher's whiteboard picture or the currently speaking student's picture to achieve enhancement effects.

Feature Advantages

• Low power consumption: After enabling 360p super resolution, taking OPPO R11 as an example, the current increment is less than 60mA, and the additional power consumption is extremely small.
• Low heat generation: After enabling 360p super resolution for half an hour, taking OPPO R11 (Snapdragon 660) as an example, the temperature rise is less than 1.5°C.
• High performance: For devices that can enable the super resolution feature, more than 95% have a CPU increment of less than 2% and a memory increment of less than 100MB.

Sample Source Code

Please refer to Download Sample Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Others/SuperResolution" directory.

Prerequisites

Before using the super resolution feature, please ensure:

• You have contacted ZEGO technical support for special packaging.
• You have integrated ZEGO Express SDK in the project and implemented basic real-time audio and video functions. For details, please refer to Quick Start.
• You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Management.

Implementation Flow

1 Initialize and login room

For the specific process of initializing and logging in to the room, please refer to "Create Engine" and "Login Room" in the implementation video call documentation.

2 Listen for super resolution status callback

After enabling the super resolution feature, you can confirm whether the super resolution feature is successfully enabled through onPlayerVideoSuperResolutionUpdate.

The super resolution feature status is defined as follows:

You can get the playing stream video resolution after the super resolution feature is enabled through onPlayerVideoSizeChanged.

// Notification of playing stream video resolution change
-(void)onPlayerVideoSizeChanged:(CGSize)size streamID:(NSString *)streamID {

}
// Notification of super resolution status change
-(void)onPlayerVideoSuperResolutionUpdate:(NSString *)streamID state:(ZegoSuperResolutionState)state errorCode:(int)errorCode {
    NSString *state_str;
    if(state == ZegoSuperResolutionStateOn)
    {
        // Super resolution is enabled
    }
    else
    {
        // Super resolution is disabled
        if(errorCode == 0){
            // Normal shutdown
        }
        else if(errorCode == 1004004){
            // The device does not support super resolution
        }
        else if(errorCode == 1004005){
            // The number of super resolution streams exceeds the limit, only supports super resolution for one stream
        }
        else if(errorCode == 1004006){
            // Super resolution original resolution exceeds the limit
        }
        else if(errorCode == 1004007){
            // Insufficient device performance for super resolution
        }
        else if(errorCode == 1004008){
            // Super resolution not initialized, please initialize super resolution first
        }
    }
}

// Notification of playing stream video resolution change
-(void)onPlayerVideoSizeChanged:(CGSize)size streamID:(NSString *)streamID {

}
// Notification of super resolution status change
-(void)onPlayerVideoSuperResolutionUpdate:(NSString *)streamID state:(ZegoSuperResolutionState)state errorCode:(int)errorCode {
    NSString *state_str;
    if(state == ZegoSuperResolutionStateOn)
    {
        // Super resolution is enabled
    }
    else
    {
        // Super resolution is disabled
        if(errorCode == 0){
            // Normal shutdown
        }
        else if(errorCode == 1004004){
            // The device does not support super resolution
        }
        else if(errorCode == 1004005){
            // The number of super resolution streams exceeds the limit, only supports super resolution for one stream
        }
        else if(errorCode == 1004006){
            // Super resolution original resolution exceeds the limit
        }
        else if(errorCode == 1004007){
            // Insufficient device performance for super resolution
        }
        else if(errorCode == 1004008){
            // Super resolution not initialized, please initialize super resolution first
        }
    }
}

3 Initialize video super resolution

Before using the video super resolution feature, you need to call the initVideoSuperResolution interface to initialize video super resolution.

[[ZegoExpressEngine sharedEngine]initVideoSuperResolution];

[[ZegoExpressEngine sharedEngine]initVideoSuperResolution];

4 Enable video super resolution

Developers can choose to enable the video super resolution feature before or during playing stream. Taking enabling the super resolution feature for stream "STREAM_ID" before playing stream as an example:

[[ZegoExpressEngine sharedEngine]enableVideoSuperResolution:@"STREAM_ID" enable:YES];

[[ZegoExpressEngine sharedEngine]enableVideoSuperResolution:@"STREAM_ID" enable:YES];

After enabling the super resolution feature, you need to listen to the callback onPlayerVideoSuperResolutionUpdate to confirm whether the super resolution feature is successfully enabled this time.

5 Start playing stream

Call startPlayingStream and pass in "STREAM_ID" to play the remote user's audio and video stream.

ZegoCanvas *playCanvas = [ZegoCanvas canvasWithView:playView];
[[ZegoExpressEngine sharedEngine]startPlayingStream:@"STREAM_ID" canvas:playCanvas];

ZegoCanvas *playCanvas = [ZegoCanvas canvasWithView:playView];
[[ZegoExpressEngine sharedEngine]startPlayingStream:@"STREAM_ID" canvas:playCanvas];

6 Disable video super resolution

When the super resolution feature is not needed or an error occurs during the super resolution feature (insufficient performance or original video resolution exceeds the limit), the super resolution feature can be disabled to release system resources.

[[ZegoExpressEngine sharedEngine]enableVideoSuperResolution:@"STREAM_ID" enable:NO];

[[ZegoExpressEngine sharedEngine]enableVideoSuperResolution:@"STREAM_ID" enable:NO];

7 Stop playing stream

After stopping playing stream, the SDK will automatically disable the super resolution feature and release occupied system resources.

[[ZegoExpressEngine sharedEngine]stopPlayingStream:@"STREAM_ID"];

[[ZegoExpressEngine sharedEngine]stopPlayingStream:@"STREAM_ID"];

8 Uninitialize video super resolution

When the video super resolution feature is not needed, you can call the uninitVideoSuperResolution interface to uninitialize video super resolution to save memory.

[[ZegoExpressEngine sharedEngine]uninitVideoSuperResolution];

[[ZegoExpressEngine sharedEngine]uninitVideoSuperResolution];