Stream Quality Monitoring

Overview

When using ZEGOCLOUD Express SDK for calls, users may experience poor network conditions. You can use the relevant callbacks to understand the current network quality and audio/video information changes during the call.

For example, in multi-person audio/video calls or multi-person singing scenarios, you may need to display user network quality in real-time. You can refer to this document to implement the corresponding functionality.

Sample Source Code

Please refer to Download Sample Source Code to get the source code.

For related source code, please check files in the "lib\topics\StreamAdvanced\stream_monitoring" directory.

Prerequisites

Before monitoring call quality, make sure:

• ZEGOCLOUD Express SDK has been integrated in your project, and basic real-time audio/video functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
• A project has been created in the ZEGOCLOUD Console, and valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.

Basic Network Quality Report

You can monitor the onNetworkQuality callback to receive the upstream and downstream network quality of users in the room (including yourself). This callback is received every two seconds. For network quality levels, please refer to ZegoStreamQualityLevel.

The callback logic varies by version:

ZegoExpressEngine.onNetworkQuality = (String userID, ZegoStreamQualityLevel upstreamQuality, ZegoStreamQualityLevel downstreamQuality) {
    if (userID.isEmpty) {
        // Represents local user's (local side) network quality
        // ("My upstream network quality is $upstreamQuality");
        // ("My downstream network quality is $downstreamQuality");
    } else {
        // Represents other users' network quality in the room
        //("User $userID's upstream network quality is $upstreamQuality");
        //("User $userID's downstream network quality is $downstreamQuality");
    }

    /*
    ZegoStreamQualityLevel.Excellent, Excellent network quality
    ZegoStreamQualityLevel.Good, Good network quality
    ZegoStreamQualityLevel.Medium, Medium network quality
    ZegoStreamQualityLevel.Bad, Poor network quality
    ZegoStreamQualityLevel.Die, Network abnormal
    ZegoStreamQualityLevel.Unknown, Unknown network quality
    */
};

ZegoExpressEngine.onNetworkQuality = (String userID, ZegoStreamQualityLevel upstreamQuality, ZegoStreamQualityLevel downstreamQuality) {
    if (userID.isEmpty) {
        // Represents local user's (local side) network quality
        // ("My upstream network quality is $upstreamQuality");
        // ("My downstream network quality is $downstreamQuality");
    } else {
        // Represents other users' network quality in the room
        //("User $userID's upstream network quality is $upstreamQuality");
        //("User $userID's downstream network quality is $downstreamQuality");
    }

    /*
    ZegoStreamQualityLevel.Excellent, Excellent network quality
    ZegoStreamQualityLevel.Good, Good network quality
    ZegoStreamQualityLevel.Medium, Medium network quality
    ZegoStreamQualityLevel.Bad, Poor network quality
    ZegoStreamQualityLevel.Die, Network abnormal
    ZegoStreamQualityLevel.Unknown, Unknown network quality
    */
};

Advanced Quality Report

If the above basic network quality report cannot meet your needs, ZEGOCLOUD also provides more detailed publishing stream quality reports, playing stream quality reports, and other related information for your reference.

Publishing Stream Quality Report

The publishing stream quality report describes the quality of the process where users push audio/video to the ZEGOCLOUD server, including the frame rate of the audio/video stream during the capture and encoding stages, and the frame rate, bitrate, latency, and packet loss rate during transmission (sending).

You can register for onPublisherQualityUpdate to receive publishing stream quality callbacks. After successfully publishing a stream, this callback will be received every three seconds. You can use the quality parameter (ZegoPublishStreamQuality) to understand the health status of the published audio/video stream in real-time.

• In most cases, you only need to pay attention to the "level" parameter of "quality" and use the "level" enumeration value to determine the overall quality of the published stream. For details, please refer to ZegoStreamQualityLevel.
• If you want to pay attention to more detailed publishing stream quality parameters, please refer to ZegoPublishStreamQuality.

// Developers can monitor specific quality in this callback and report to the business server for monitoring, or monitor specific fields of the quality object to provide user-friendly prompts
ZegoExpressEngine.onPublisherQualityUpdate = (String streamID, ZegoPublishStreamQuality quality) {
    String networkQuality = "";
    // level represents the comprehensive score of publishing stream quality. In most cases, developers can refer to this score to display upstream network quality

    switch (quality.level) {
        case ZegoStreamQualityLevel.Excellent:
           networkQuality = "Excellent";
           break;
        case ZegoStreamQualityLevel.Good:
           networkQuality = "Good";
           break;
        case ZegoStreamQualityLevel.Medium:
           networkQuality = "Medium";
           break;
        case ZegoStreamQualityLevel.Bad:
           networkQuality = "Poor";
           break;
        case ZegoStreamQualityLevel.Die:
           networkQuality = "Failed";
           break;
        case ZegoStreamQualityLevel.Unknown:
           networkQuality = "Unknown";
           break;
        default:
           break;
    }
    //("Network quality is: $networkQuality");
};

// Developers can monitor specific quality in this callback and report to the business server for monitoring, or monitor specific fields of the quality object to provide user-friendly prompts
ZegoExpressEngine.onPublisherQualityUpdate = (String streamID, ZegoPublishStreamQuality quality) {
    String networkQuality = "";
    // level represents the comprehensive score of publishing stream quality. In most cases, developers can refer to this score to display upstream network quality

    switch (quality.level) {
        case ZegoStreamQualityLevel.Excellent:
           networkQuality = "Excellent";
           break;
        case ZegoStreamQualityLevel.Good:
           networkQuality = "Good";
           break;
        case ZegoStreamQualityLevel.Medium:
           networkQuality = "Medium";
           break;
        case ZegoStreamQualityLevel.Bad:
           networkQuality = "Poor";
           break;
        case ZegoStreamQualityLevel.Die:
           networkQuality = "Failed";
           break;
        case ZegoStreamQualityLevel.Unknown:
           networkQuality = "Unknown";
           break;
        default:
           break;
    }
    //("Network quality is: $networkQuality");
};

Playing Stream Quality Report

The playing stream quality report describes the quality of the process where users play audio/video streams, including the frame rate, bitrate, latency, and packet loss rate of the received audio/video stream, the frame rate during the decoding stage, and the frame rate, freeze rate, and overall audio/video quality during the rendering stage.

You can register for onPlayerQualityUpdate to receive playing stream quality callbacks. After successfully playing a stream, this callback will be received every three seconds. You can use the quality parameter (ZegoPlayStreamQuality) to understand the health status of the played audio/video stream in real-time.

• In most cases, you only need to pay attention to the "level" parameter of "quality" and use the "level" enumeration value to determine the overall quality of the played stream. For details, please refer to ZegoStreamQualityLevel.
• If you want to pay attention to more detailed playing stream quality parameters, please refer to ZegoPlayStreamQuality.

// Developers can monitor specific quality in this callback and report to the business server for monitoring, or monitor specific fields of the quality object to provide user-friendly prompts
ZegoExpressEngine.onPlayerQualityUpdate = (String streamID, ZegoPlayStreamQuality quality) {
    String networkQuality = "";
    // level represents the comprehensive score of playing stream quality. In most cases, developers can refer to this score to display downstream network quality

    switch (quality.level) {
        case ZegoStreamQualityLevel.Excellent:
           networkQuality = "Excellent";
           break;
        case ZegoStreamQualityLevel.Good:
           networkQuality = "Good";
           break;
        case ZegoStreamQualityLevel.Medium:
           networkQuality = "Medium";
           break;
        case ZegoStreamQualityLevel.Bad:
           networkQuality = "Poor";
           break;
        case ZegoStreamQualityLevel.Die:
           networkQuality = "Failed";
           break;
        case ZegoStreamQualityLevel.Unknown:
           networkQuality = "Unknown";
           break;
        default:
           break;
    }
    //("Network quality is: %s", networkQuality);
};

// Developers can monitor specific quality in this callback and report to the business server for monitoring, or monitor specific fields of the quality object to provide user-friendly prompts
ZegoExpressEngine.onPlayerQualityUpdate = (String streamID, ZegoPlayStreamQuality quality) {
    String networkQuality = "";
    // level represents the comprehensive score of playing stream quality. In most cases, developers can refer to this score to display downstream network quality

    switch (quality.level) {
        case ZegoStreamQualityLevel.Excellent:
           networkQuality = "Excellent";
           break;
        case ZegoStreamQualityLevel.Good:
           networkQuality = "Good";
           break;
        case ZegoStreamQualityLevel.Medium:
           networkQuality = "Medium";
           break;
        case ZegoStreamQualityLevel.Bad:
           networkQuality = "Poor";
           break;
        case ZegoStreamQualityLevel.Die:
           networkQuality = "Failed";
           break;
        case ZegoStreamQualityLevel.Unknown:
           networkQuality = "Unknown";
           break;
        default:
           break;
    }
    //("Network quality is: %s", networkQuality);
};

MOS Audio Quality Score

Starting from ZEGOCLOUD Express SDK version 2.16.0, the playing stream quality callback onPlayerQualityUpdate includes a new "mos" field, representing the score of the played stream's audio quality. Developers who are concerned about audio quality can use this field to understand the current audio quality status.

The "mos" field value ranges from [-1, 5], where -1 means unknown (for example, when scoring is not possible due to abnormal stream playing), and [0, 5] represents the normal scoring range. The subjective audio quality perception corresponding to real-time audio MOS scoring is as follows:

Other Information Monitoring

Publish/Play Stream Status Change Notification

Publishing Stream Status Callback

After successfully publishing a stream, you can use onPublisherStateUpdate to receive notifications of publishing stream status changes.

ZegoExpressEngine.onPublisherStateUpdate = (String streamID, ZegoPublisherState state, int errorCode, Map<String, dynamic> extendedData) {
    // When state is ZegoPublisherState.NoPublish and errorCode is non-zero, it indicates publishing stream failure, and no more retries will be made. At this point, you can display a publishing stream failure prompt on the interface
    // When state is ZegoPublisherState.PublishRequesting and errorCode is non-zero, it indicates retrying to publish stream. If the stream is not successfully published within the retry time, a publishing stream failure notification will be thrown
};

ZegoExpressEngine.onPublisherStateUpdate = (String streamID, ZegoPublisherState state, int errorCode, Map<String, dynamic> extendedData) {
    // When state is ZegoPublisherState.NoPublish and errorCode is non-zero, it indicates publishing stream failure, and no more retries will be made. At this point, you can display a publishing stream failure prompt on the interface
    // When state is ZegoPublisherState.PublishRequesting and errorCode is non-zero, it indicates retrying to publish stream. If the stream is not successfully published within the retry time, a publishing stream failure notification will be thrown
};

You can roughly judge the user's publishing network situation based on whether the "state" parameter in the callback is in "requesting publish stream status". The values of the "state" parameter correspond to the user's publishing stream status as follows:

The parameter "extendedData" is the extended information attached to the status update. If using ZEGOCLOUD's CDN content distribution network, after successfully publishing the stream, the content of this parameter includes keys "flv_url_list", "rtmp_url_list", "hls_url_list", corresponding to the playing URLs for FLV, RTMP, and HLS protocols respectively.

Playing Stream Status Change Callback

After successfully playing a stream, developers can use onPlayerStateUpdate to receive notifications of playing stream status changes.

ZegoExpressEngine.onPlayerStateUpdate = (String streamID, ZegoPlayerState state, int errorCode, Map<String, dynamic>extendedData) {
    // When state is ZegoPlayerState.NoPlay and errorCode is non-zero, it indicates playing stream failure, and no more retries will be made. At this point, you can display a playing stream failure prompt on the interface
    // When state is ZegoPlayerState.PlayRequesting and errorCode is non-zero, it indicates retrying to play stream. If the stream is not successfully played within the retry time, a playing stream failure notification will be thrown
};

ZegoExpressEngine.onPlayerStateUpdate = (String streamID, ZegoPlayerState state, int errorCode, Map<String, dynamic>extendedData) {
    // When state is ZegoPlayerState.NoPlay and errorCode is non-zero, it indicates playing stream failure, and no more retries will be made. At this point, you can display a playing stream failure prompt on the interface
    // When state is ZegoPlayerState.PlayRequesting and errorCode is non-zero, it indicates retrying to play stream. If the stream is not successfully played within the retry time, a playing stream failure notification will be thrown
};

Developers can roughly judge the user's playing network situation based on whether the "state" parameter is in "requesting play stream status". The values of the "state" parameter correspond to the user's playing stream status as follows:

Notification of Receiving Audio/Video First Frame

Publishing Side Audio Capture First Frame Callback

You can register for onPublisherCapturedAudioFirstFrame to receive the audio first frame callback. After successfully calling the publish stream interface, this callback will be received when the SDK captures the first frame of audio data.

ZegoExpressEngine.onPublisherCapturedAudioFirstFrame = () {
};

ZegoExpressEngine.onPublisherCapturedAudioFirstFrame = () {
};

Publishing Side Video Capture First Frame Callback

You can register for onPublisherCapturedVideoFirstFrame to receive the video first frame callback. After successfully calling the publish stream interface, this callback will be received when the SDK captures the first frame of video data.

ZegoExpressEngine.onPublisherCapturedVideoFirstFrame = (ZegoPublishChannel channel) {
};

ZegoExpressEngine.onPublisherCapturedVideoFirstFrame = (ZegoPublishChannel channel) {
};

Playing Side Audio Receive First Frame Callback

Developers can register for onPlayerRecvAudioFirstFrame to monitor the playing side's audio receive first frame callback. After successfully calling the play stream interface, this callback will be received when the SDK plays the first frame of audio data.

ZegoExpressEngine.onPlayerRecvAudioFirstFrame = (String streamID) {
    print("onPlayerRecvAudioFirstFrame streamID:$streamID");
};

ZegoExpressEngine.onPlayerRecvAudioFirstFrame = (String streamID) {
    print("onPlayerRecvAudioFirstFrame streamID:$streamID");
};

Playing Side Video Receive First Frame Callback

You can register for onPlayerRecvVideoFirstFrame to monitor the playing side's video receive first frame callback. After successfully calling the play stream interface, this callback will be received when the SDK plays the first frame of video data.

ZegoExpressEngine.onPlayerRecvVideoFirstFrame = (String streamID) {
};

ZegoExpressEngine.onPlayerRecvVideoFirstFrame = (String streamID) {
};

Playing Side Rendered Video First Frame Callback

You can register for onPlayerRenderVideoFirstFrame to monitor the playing side's rendered video first frame callback. After successfully calling the play stream interface, this callback will be received when the SDK plays and renders the first frame of video data.

ZegoExpressEngine.onPlayerRenderVideoFirstFrame = (String streamID) {
};

ZegoExpressEngine.onPlayerRenderVideoFirstFrame = (String streamID) {
};

Video Resolution Change Callback

Capture Video Resolution Change Callback

You can register for onPublisherVideoSizeChanged to monitor the capture video size change callback. After successfully publishing a stream, if the video capture resolution changes during publishing, this callback will be received.

ZegoExpressEngine.onPublisherVideoSizeChanged = (int width, int height, ZegoPublishChannel channel) {
};

ZegoExpressEngine.onPublisherVideoSizeChanged = (int width, int height, ZegoPublishChannel channel) {
};

Playing Resolution Change Notification

You can register for onPlayerVideoSizeChanged to receive playing resolution change notifications. After successfully playing a stream, if the video resolution changes during playing, this callback will be received. Users can adjust the display based on the stream's final resolution.

ZegoExpressEngine.onPlayerVideoSizeChanged = (String streamID, int width, int height) {
};

ZegoExpressEngine.onPlayerVideoSizeChanged = (String streamID, int width, int height) {
};

Related Documentation

• How to handle video freezing issues?
• How to handle audio freezing issues?
• How to understand and use SEI (Supplemental Enhancement Information)?