Call Quality Monitoring

Feature Overview

During a call or live streaming, users can obtain call-related information by registering relevant callbacks, including quality reports for publishing and playing streams, callbacks for receiving the first audio and video frame, resolution changes, CDN (Content Delivery Network) related information, and receiving SEI (Supplemental Enhancement Information). This article will introduce the following features:

By obtaining the above information, developers can perform related operations based on publishing/playing status. For example, take corresponding actions when publishing quality is poor, determine whether publishing/playing is successful, determine whether the audio and video streams relayed to CDN are normal, or receive SEI information.

Related concept explanations:

• Publishing stream: refers to the process of transmitting the packaged content from the capture stage to the server.
• Playing stream: refers to the process of pulling existing live content from the server using a specified address.

Sample Source Code Download

Please refer to Download Sample Source Code to obtain the source code.

For related source code, please check the files in the "/ZegoExpressExample/Examples/AdvancedStreaming/StreamMonitoring" directory.

Prerequisites

Before performing call quality monitoring, please ensure:

• A project has been created in the ZEGOCLOUD Console, and valid AppID and AppSign have been obtained. For details, please refer to Console - Project Information.
• ZEGO Express SDK has been integrated into the project, and basic audio/video publishing and playing functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Implementation Methods

User Network Quality

You can receive the upstream and downstream network quality of users (including yourself) by listening to the onNetworkQuality callback. This callback is received every two seconds. For network quality levels, please refer to ZegoStreamQualityLevel.

The callback logic of onNetworkQuality varies by version:

class MyEventHandler : public IZegoEventHandler{
public:
    void onNetworkQuality(const std::string& userID, ZegoStreamQualityLevel upstreamQuality, ZegoStreamQualityLevel downstreamQuality){
        // Developers can monitor the user's upstream and downstream network quality in this callback to report to the business server for monitoring, or give user-friendly prompts
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onNetworkQuality(const std::string& userID, ZegoStreamQualityLevel upstreamQuality, ZegoStreamQualityLevel downstreamQuality){
        // Developers can monitor the user's upstream and downstream network quality in this callback to report to the business server for monitoring, or give user-friendly prompts
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Monitor Publishing Quality

Developers can receive publishing quality callbacks by registering onPublisherQualityUpdate. After successful publishing, this callback will be received every three seconds. Developers can monitor the health status of published audio and video streams in real-time based on quality parameters returned by the callback, so as to display upstream network conditions on the device UI in real-time.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherQualityUpdate(String streamID, ZegoPublishStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor certain fields of the quality object to give user-friendly prompts
        // If developers don't know which quality field to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherQualityUpdate(String streamID, ZegoPublishStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor certain fields of the quality object to give user-friendly prompts
        // If developers don't know which quality field to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Publishing Quality Details

Publishing quality includes the frame rates of audio and video streams in the capture and encoding stages, and the frame rate, bitrate, latency, and packet loss rate of audio and video streams during transmission (sending). This section will introduce the parameters in publishing quality ZegoPublishStreamQuality.

Publishing Capture Quality

Publishing capture quality is close to the user's subjective feeling during preview. The relevant parameters of audio and video quality in the capture stage during publishing are as follows:

• audioCaptureFPS: Audio capture frame rate (fps)
• videoCaptureFPS: Video capture frame rate (fps)

Publishing Encoding Quality

The relevant parameters of audio and video quality in the encoding stage during publishing are as follows:

• videoEncodeFPS: Current encoder's target video encoding frame rate (fps)

Publishing Send Quality

Publishing send quality is the actual publishing quality, related to actual network quality. The relevant parameters are as follows:

• audioSendFPS: Actual audio send frame rate (fps)
• audioKBPS: Actual audio send bitrate (kbps)
• videoSendFPS: Actual video send frame rate (fps)
• videoKBPS: Actual video send bitrate (kbps)
• rtt: Round-trip latency from device to ZEGO Server (ms)
• packetLostRate: Device upstream packet loss rate

Byte Count Statistics

Developers can count the total bytes of sent video, audio, and total. The relevant parameters are as follows:

• totalSendBytes: Total bytes sent
• audioSendBytes: Audio bytes sent
• videoSendBytes: Video bytes sent

Encoding Information

Developers can obtain the encoding information of published streams. The relevant parameters are as follows:

• videoCodecID: Video encoding format
• isHardwareEncode: Whether hardware encoding is enabled

Upstream Network Comprehensive Quality

If developers are not sure how to use each parameter of this callback interface, they can focus only on the level parameter, which is a comprehensive value describing upstream network calculated internally by ZegoExpressEngine based on quality parameters.

The description of the level field is as follows:

Monitor Playing Quality

Developers can receive playing quality callbacks by registering onPlayerQualityUpdate. After successful playing, this callback will be received every three seconds. Developers can monitor the health status of played audio and video streams in real-time based on quality parameters returned by the callback, so as to display downstream network conditions on the device UI in real-time.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerQualityUpdate(String streamID, ZegoPlayStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor certain fields of the quality object to give user-friendly prompts
        // If developers don't know which quality field to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerQualityUpdate(String streamID, ZegoPlayStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor certain fields of the quality object to give user-friendly prompts
        // If developers don't know which quality field to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing Quality Details

Playing quality includes the frame rate, bitrate, latency, and packet loss rate of received audio and video streams, the frame rate of audio and video streams in the decoding stage, and the frame rate, freeze rate, and overall audio and video quality in the rendering stage. This section will introduce the parameters in playing quality ZegoPlayStreamQuality.

Playing Receive Quality

Playing receive quality is the actual playing quality, related to actual publishing quality and current network quality. The relevant parameters are as follows:

• audioRecvFPS: Actual received audio frame rate (fps)
• audioDejitterFPS: Audio dejitter frame rate (f/s)
• audioKBPS: Actual received audio bitrate (kbps)
• audioBreakRate: Actual received audio freeze rate (freeze count per 10 seconds)
• videoRecvFPS: Actual received video frame rate (fps)
• videoDejitterFPS: Video dejitter frame rate (f/s)
• videoKBPS: Actual received video bitrate (kbps)
• videoBreakRate: Actual received video freeze rate (freeze count per 10 seconds)
• packetLostRate: Device downstream packet loss rate
• rtt: Round-trip latency from device to ZEGO Server (ms)
• avTimestampDiff: Difference between video timestamp and audio timestamp, used to reflect audio-video synchronization status, in milliseconds. A value less than 0 indicates the number of milliseconds video is ahead of audio, greater than 0 indicates the number of milliseconds video lags behind audio, equal to 0 indicates no difference. When the absolute value is less than 200, it can be basically considered that audio and video are synchronized. When the absolute value is continuously greater than 200 for 10 seconds, it can be considered abnormal
• packetLostRate: Packet loss rate, in percentage, with a value range of 0.0 ~ 1.0
• peerToPeerDelay: End-to-end latency, in milliseconds
• peerToPeerPacketLostRate: End-to-end packet loss rate, in percentage, with a value range of 0.0 ~ 1.0

Playing Render Quality

Playing render quality is close to the user's subjective feeling when watching audio and video. This quality may be lower than the actual received playing quality value due to the influence of the decoder. The relevant parameters are as follows:

• audioRenderFPS: Actual audio render frame rate
• videoRenderFPS: Actual video render frame rate
• delay: Latency from local receiving data to playing, in milliseconds

Byte Count Statistics

Developers can count the total bytes of received video, audio, and total. The relevant parameters are as follows:

• totalRecvBytes: Total bytes received, including audio, video, SEI, etc.
• audioRecvBytes: Audio bytes received
• videoRecvBytes: Video bytes received

Decoding Information

Developers can obtain the decoding information of played streams. The relevant parameters are as follows:

• videoCodecID: Video decoding format
• isHardwareDecode: Whether hardware decoding is enabled

MOS Audio Quality Score

Starting from ZEGO Express SDK version 2.16.0, a new mos field has been added to the playing quality callback onPlayerQualityUpdate, representing a score for playing audio quality. When developers are concerned about audio quality, they can use this field to understand the current audio quality status.

The value range of the mos field is [-1, 5], where -1 indicates unknown (for example, unable to score when playing abnormally), and [0, 5] indicates the normal scoring range. The subjective audio quality feelings corresponding to real-time audio MOS scores are as follows:

Monitor Publishing/Playing Status

Publishing Status Callback

After successful publishing, developers can obtain notifications of publishing status changes through onPublisherStateUpdate.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherStateUpdate(String streamID, ZegoPublisherState state, int errorCode, JSONObject extendedData){
    // When state is PUBLISHER_STATE_NO_PUBLISH, and errcode is non-zero, it indicates publishing failure, and no more retry publishing will be performed. At this time, a publishing failure prompt can be displayed on the interface;
    // When state is PUBLISHER_STATE_PUBLISH_REQUESTING, and errcode is non-zero, it indicates retrying publishing. If publishing is not successful within the retry time, a publishing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherStateUpdate(String streamID, ZegoPublisherState state, int errorCode, JSONObject extendedData){
    // When state is PUBLISHER_STATE_NO_PUBLISH, and errcode is non-zero, it indicates publishing failure, and no more retry publishing will be performed. At this time, a publishing failure prompt can be displayed on the interface;
    // When state is PUBLISHER_STATE_PUBLISH_REQUESTING, and errcode is non-zero, it indicates retrying publishing. If publishing is not successful within the retry time, a publishing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Developers can roughly determine the user's publishing network situation based on whether the "state" parameter in the callback is in "requesting publishing status". The values of the "state" parameter correspond to user publishing status as follows:

The parameter "extendedData" is extended information attached to the status update. If using ZEGO's CDN content delivery network, after successful publishing, the keys of the content of this parameter are "flv_url_list", "rtmp_url_list", "hls_url_list", corresponding to the playing URLs of flv, rtmp, and hls protocols respectively.

Playing Status Change Callback

After successful playing, developers can obtain notifications of playing status changes through onPlayerStateUpdate.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerStateUpdate(String streamID, ZegoPlayerState state, int errorCode, JSONObject extendedData){
        // When state is PLAYER_STATE_NO_PLAY, and errcode is non-zero, it indicates playing failure, and no more retry playing will be performed. At this time, a playing failure prompt can be displayed on the interface;
        // When state is PLAYER_STATE_PLAY_REQUESTING, and errcode is non-zero, it indicates retrying playing. If playing is not successful within the retry time, a playing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerStateUpdate(String streamID, ZegoPlayerState state, int errorCode, JSONObject extendedData){
        // When state is PLAYER_STATE_NO_PLAY, and errcode is non-zero, it indicates playing failure, and no more retry playing will be performed. At this time, a playing failure prompt can be displayed on the interface;
        // When state is PLAYER_STATE_PLAY_REQUESTING, and errcode is non-zero, it indicates retrying playing. If playing is not successful within the retry time, a playing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Developers can roughly determine the user's playing network situation based on whether the "state" parameter is in "requesting playing status". The values of the "state" parameter correspond to user playing status as follows:

Audio and Video First Frame Callback

Publishing Audio Capture First Frame Callback

Developers can receive audio first frame callbacks by registering onPublisherCapturedAudioFirstFrame. After successfully calling the publishing interface, this callback will be received when the SDK captures the first frame of audio data.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedAudioFirstFrame(){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedAudioFirstFrame(){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Publishing Video Capture First Frame Callback

Developers can receive video first frame callbacks by registering onPublisherCapturedVideoFirstFrame. After successfully calling the publishing interface, this callback will be received when the SDK captures the first frame of video data.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedVideoFirstFrame(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedVideoFirstFrame(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing Audio Receive First Frame Callback

Developers can monitor the playing audio receive first frame callback by registering onPlayerRecvAudioFirstFrame. After successfully calling the playing interface, this callback will be received when the SDK plays and receives the first frame of audio data.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvAudioFirstFrame(String streamID){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvAudioFirstFrame(String streamID){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing Video Receive First Frame Callback

Developers can monitor the playing video receive first frame callback by registering onPlayerRecvVideoFirstFrame. After successfully calling the playing interface, this callback will be received when the SDK plays and receives the first frame of video data.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvVideoFirstFrame(String streamID)(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvVideoFirstFrame(String streamID)(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing Render Video First Frame Callback

Developers can monitor the playing render video first frame callback by registering onPlayerRenderVideoFirstFrame. After successfully calling the playing interface, this callback will be received when the SDK plays and renders the first frame of video data.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRenderVideoFirstFrame{

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRenderVideoFirstFrame{

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Monitor Video Resolution Changes

Capture Video Resolution Change Callback

Developers can monitor the capture video size change callback by registering onPublisherVideoSizeChanged. After successful publishing, if the video capture resolution changes during publishing, this callback will be received.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherVideoSizeChanged(int width, int height, ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherVideoSizeChanged(int width, int height, ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing Resolution Change Notification

Developers can obtain playing resolution change notifications by registering onPlayerVideoSizeChanged. After successful playing, if the video resolution changes during playing, this callback will be received. Users can adjust the display based on the final resolution of the stream.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerVideoSizeChanged(String streamID, int width, int height){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerVideoSizeChanged(String streamID, int width, int height){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Monitor CDN Relay Status

Add/Remove Relay CDN Address Status Callback

Developers can obtain add/remove relay CDN address status callbacks by registering onPublisherRelayCDNStateUpdate. After ZEGO RTC server relays audio and video streams to CDN, if the CDN relay status changes, such as relay stopping or relay retrying, this callback will be received.

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherRelayCDNStateUpdate(String streamID, ArrayList<ZegoStreamRelayCDNInfo> infoList){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherRelayCDNStateUpdate(String streamID, ArrayList<ZegoStreamRelayCDNInfo> infoList){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Relay CDN Information Details

Relay CDN information ZegoStreamRelayCDNInfo includes the URL of CDN publishing, relay status, reason for relay status change, and time when the status occurred. All parameters in ZegoStreamRelayCDNInfo are as follows:

Among them, state values are as follows:

updateReason values are as follows:

Related Documents

• How to handle video freeze issues?
• How to handle audio freeze issues?
• How to understand and use SEI (Supplemental Enhancement Information)?