Publish/Play Stream Monitoring

Feature Overview

During the stream publishing and playing process, users can obtain information about published and played streams by registering relevant callbacks, including publishing/playing stream quality, first frame callbacks, resolution changes, CDN (Content Delivery Network) related information, and receiving SEI (Supplemental Enhancement Information). This document will introduce the following six features:

By obtaining the above information, developers can perform related operations based on the publishing/playing stream state. For example, make corresponding handling when publishing stream quality is poor, determine whether publishing/playing stream is successful, determine whether the audio and video streams relayed to CDN are normal, or receive SEI information, etc.

Related concept explanation:

• Publish stream: Refers to the process of transmitting the content packaged during the capture stage to the server.
• Play stream: Refers to the process of pulling existing live content from the server using a specified address.

Prerequisites

Before monitoring publishing and playing stream information, ensure:

• ZEGO Express SDK has been integrated into the project and basic real-time Video Call functions have been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.
• A project has been created in the ZEGOCLOUD Console and a valid AppID and AppSign have been applied. For details, please refer to Console - Project Management.

Example Source Code Download

Please refer to Download Example Source Code to obtain the source code.

For related source code, please check the files in the "/ZegoExpressExample/AdvancedStreaming/src/main/java/im/zego/streammonitoring" directory.

Usage Steps

Monitor publishing stream quality

Developers can receive publishing stream quality callbacks by registering onPublisherQualityUpdate. After successful publishing, this callback will be received every three seconds. Developers can monitor the health of the published audio and video streams in real-time based on the quality parameters returned by the callback to display the uplink network status in real-time on the device UI interface.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherQualityUpdate(String streamID, ZegoPublishStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor a certain field of the quality object to give user-friendly prompts
        // If developers don't know which field of the quality to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherQualityUpdate(String streamID, ZegoPublishStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor a certain field of the quality object to give user-friendly prompts
        // If developers don't know which field of the quality to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Publishing stream quality details

Publishing stream quality includes the frame rate and resolution of audio and video streams during the capture and encoding stages, as well as the frame rate, bitrate, latency, and packet loss rate of the audio and video streams during transmission (sending). This section will introduce the parameters within the publishing stream quality ZegoPublishStreamQuality.

Publishing capture quality

Publishing capture quality is close to the user's subjective experience during preview. The relevant parameters of the audio and video quality during the capture stage when publishing are as follows:

• audioCaptureFPS: Audio capture frame rate (fps)
• videoCaptureFPS: Video capture frame rate (fps)

Publishing encoding quality

The relevant parameters of the audio and video quality during the encoding stage when publishing are as follows:

• videoEncodeFPS: The target video encoding frame rate of the current encoder (fps)

Publishing sending quality

Publishing sending quality is the actual publishing quality, which is related to the set encoding resolution and actual network quality. The relevant parameters are as follows:

• audioSendFPS: Actual audio sending frame rate (fps)
• audioKBPS: Actual audio sending bitrate (kbps)
• videoSendFPS: Actual video sending frame rate (fps)
• videoKBPS: Actual video sending bitrate (kbps)
• rtt: Round-trip time (RTT) from device to ZEGO Server (ms)
• packetLostRate: Device uplink packet loss rate

Byte count statistics

Developers can count the total bytes of sent video, audio, and total bytes. The relevant parameters are as follows:

• totalSendBytes: Total bytes sent
• audioSendBytes: Audio bytes sent
• videoSendBytes: Video bytes sent

Encoding information

Developers can obtain the encoding information of the published stream. The relevant parameters are as follows:

• videoCodecID: Video encoding format
• isHardwareEncode: Whether hardware encoding is enabled

Uplink network comprehensive quality

If publishers are unclear about how to use each parameter of this callback interface, they can focus on the level parameter, which is a comprehensive value describing the uplink network calculated internally by ZegoExpressEngine based on quality parameters.

The description of the level field is as follows:

Monitor playing stream quality

Developers can receive playing stream quality callbacks by registering onPlayerQualityUpdate. After successful playing, this callback will be received every three seconds. Developers can monitor the health of the played audio and video streams in real-time based on the quality parameters returned by the callback to display the downlink network status in real-time on the device UI interface.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerQualityUpdate(String streamID, ZegoPlayStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor a certain field of the quality object to give user-friendly prompts
        // If developers don't know which field of the quality to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerQualityUpdate(String streamID, ZegoPlayStreamQuality quality){
        // Developers can monitor specific quality in this callback to report to the business server for monitoring, or monitor a certain field of the quality object to give user-friendly prompts
        // If developers don't know which field of the quality to monitor, they can focus on the level field, which is the comprehensive value of the quality object
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing stream quality details

Playing stream quality includes the frame rate, bitrate, latency, and packet loss rate of received audio and video streams, the frame rate and resolution of audio and video streams during the decoding stage, as well as the frame rate, resolution, freeze rate, and overall audio and video quality during the rendering stage. This section will introduce the parameters within the playing stream quality ZegoPlayStreamQuality.

Playing receiving quality

Playing receiving quality is the actual playing quality, which is related to the actual publishing stream quality and current network quality. The relevant parameters are as follows:

• audioRecvFPS: Actual received audio frame rate (fps)
• audioDejitterFPS: Audio dejitter frame rate, unit is f/s
• audioKBPS: Actual received audio bitrate (kbps)
• audioBreakRate: Actual received audio freeze rate (freeze count/10 seconds)
• videoRecvFPS: Actual received video frame rate (fps)
• videoDejitterFPS: Video dejitter frame rate, unit is f/s
• videoKBPS: Actual received video bitrate (kbps)
• videoBreakRate: Actual received video freeze rate (freeze count/10 seconds)
• packetLostRate: Device downlink packet loss rate
• rtt: Round-trip time (RTT) from device to ZEGO Server (ms)
• avTimestampDiff: The difference between the video timestamp and the audio timestamp, used to reflect audio-video synchronization, unit is milliseconds. A value less than 0 indicates the number of milliseconds the video is ahead of the audio, greater than 0 indicates the number of milliseconds the video lags behind the audio, equal to 0 means no difference. When the absolute value is less than 200, it can be considered that audio and video are synchronized. When the absolute value is continuously greater than 200 for 10 seconds, it can be considered abnormal
• packetLostRate: Packet loss rate, unit is percentage, 0.0 ~ 1.0
• peerToPeerDelay: Peer-to-peer delay, unit is milliseconds
• peerToPeerPacketLostRate: Peer-to-peer packet loss rate, unit is percentage, 0.0 ~ 1.0

Playing rendering quality

Playing rendering quality is close to the user's subjective experience of watching audio and video. This quality may be lower than the actual received playing stream quality due to decoder influence. The relevant parameters are as follows:

• audioRenderFPS: Actual audio rendering frame rate
• videoRenderFPS: Actual video rendering frame rate
• delay: The delay from when the local end receives data to playback, unit is milliseconds

Byte count statistics

Developers can count the total bytes of received video, audio, and total bytes. The relevant parameters are as follows:

• totalRecvBytes: Total bytes received, including audio, video, and SEI
• audioRecvBytes: Audio bytes received
• videoRecvBytes: Video bytes received

Decoding information

Developers can obtain the decoding information of the played stream. The relevant parameters are as follows:

• videoCodecID: Video decoding format
• isHardwareDecode: Whether hardware decoding is enabled

Monitor publishing/playing stream state

Publishing state callback

After successful publishing, developers can receive notifications of publishing state changes through onPublisherStateUpdate.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherStateUpdate(String streamID, ZegoPublisherState state, int errorCode, JSONObject extendedData){
    // When state is PUBLISHER_STATE_NO_PUBLISH and errcode is not 0, it indicates publishing failed and will not retry publishing. At this time, a publishing failure prompt can be displayed on the interface;
    // When state is PUBLISHER_STATE_PUBLISH_REQUESTING and errcode is not 0, it indicates retrying publishing. If publishing is not successful within the retry time, a publishing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherStateUpdate(String streamID, ZegoPublisherState state, int errorCode, JSONObject extendedData){
    // When state is PUBLISHER_STATE_NO_PUBLISH and errcode is not 0, it indicates publishing failed and will not retry publishing. At this time, a publishing failure prompt can be displayed on the interface;
    // When state is PUBLISHER_STATE_PUBLISH_REQUESTING and errcode is not 0, it indicates retrying publishing. If publishing is not successful within the retry time, a publishing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Developers can roughly judge the user's publishing network situation based on whether the "state" parameter in the callback is in "requesting to publish stream state". The values of the "state" parameter correspond to the user's publishing state as follows:

The parameter "extendedData" is extended information attached to the state update. If using ZEGO's CDN content delivery network, after successful publishing, the keys of the content of this parameter are "flv_url_list", "rtmp_url_list", "hls_url_list". These correspond to the playing URLs for flv, rtmp, and hls protocols.

Playing state change callback

After successful playing, developers can receive notifications of publishing state changes through onPlayerStateUpdate.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerStateUpdate(String streamID, ZegoPlayerState state, int errorCode, JSONObject extendedData){
        // When state is PLAYER_STATE_NO_PLAY and errcode is not 0, it indicates playing failed and will not retry playing. At this time, a playing failure prompt can be displayed on the interface;
        // When state is PLAYER_STATE_PLAY_REQUESTING and errcode is not 0, it indicates retrying playing. If playing is not successful within the retry time, a playing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerStateUpdate(String streamID, ZegoPlayerState state, int errorCode, JSONObject extendedData){
        // When state is PLAYER_STATE_NO_PLAY and errcode is not 0, it indicates playing failed and will not retry playing. At this time, a playing failure prompt can be displayed on the interface;
        // When state is PLAYER_STATE_PLAY_REQUESTING and errcode is not 0, it indicates retrying playing. If playing is not successful within the retry time, a playing failure notification will be thrown.
    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Developers can roughly judge the user's playing network situation based on whether the "state" parameter is in "requesting to play stream state". The values of the "state" parameter correspond to the user's playing state as follows:

Audio/Video first frame callback

Publisher audio capture first frame callback

Developers can receive audio first frame callbacks by registering onPublisherCapturedAudioFirstFrame. After calling the publishing interface successfully, this callback will be received when the SDK captures the first frame of audio data.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedAudioFirstFrame(){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedAudioFirstFrame(){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Publisher video capture first frame callback

Developers can receive video first frame callbacks by registering onPublisherCapturedVideoFirstFrame. After calling the publishing interface successfully, this callback will be received when the SDK captures the first frame of video data.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedVideoFirstFrame(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherCapturedVideoFirstFrame(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Player audio receive first frame callback

Developers can listen for the playing audio receive first frame callback by registering onPlayerRecvAudioFirstFrame. After calling the playing interface successfully, this callback will be received when the SDK plays the first frame of audio data.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvAudioFirstFrame(String streamID){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvAudioFirstFrame(String streamID){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Player video receive first frame callback

Developers can listen for the playing receive video first frame callback by registering onPlayerRecvVideoFirstFrame. After calling the playing interface successfully, this callback will be received when the SDK plays the first frame of video data.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvVideoFirstFrame(String streamID)(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRecvVideoFirstFrame(String streamID)(ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Player rendered video first frame callback

Developers can listen for the playing rendered video first frame callback by registering onPlayerRenderVideoFirstFrame. After calling the playing interface successfully, this callback will be received when the SDK plays and renders the first frame of video data.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRenderVideoFirstFrame{

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerRenderVideoFirstFrame{

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Monitor video resolution changes

Capture video resolution change callback

Developers can listen for the capture video size change callback by registering onPublisherVideoSizeChanged. After successful publishing, if the video capture resolution changes during publishing, this callback will be received.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherVideoSizeChanged(int width, int height, ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherVideoSizeChanged(int width, int height, ZegoPublishChannel channel){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Playing resolution change notification

Developers can receive playing resolution change notifications by registering onPlayerVideoSizeChanged. After successful playing, if the video resolution changes during playing, this callback will be received, and users can adjust the display based on the final resolution of the stream.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerVideoSizeChanged(String streamID, int width, int height){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPlayerVideoSizeChanged(String streamID, int width, int height){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Monitor CDN relay status

Add/Remove relay CDN address state callback

Developers can receive add/remove relay CDN address state callbacks by registering onPublisherRelayCDNStateUpdate. After the ZEGO RTC server relays audio and video streams to CDN, if the CDN relay status changes, such as relay stopping or relay retrying, this callback will be received.

Call example:

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherRelayCDNStateUpdate(String streamID, ArrayList<ZegoStreamRelayCDNInfo> infoList){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

class MyEventHandler : public IZegoEventHandler{
public:
    void onPublisherRelayCDNStateUpdate(String streamID, ArrayList<ZegoStreamRelayCDNInfo> infoList){

    }
}
auto eventhandler = std::make_shared<MyEventHandler>();
engine->setEventHandler(eventhandler);

Relay CDN information details

Relay CDN information ZegoStreamRelayCDNInfo contains the URL of the CDN publishing stream, relay status, reason for relay status change, and time when the status occurred. All parameters within ZegoStreamRelayCDNInfo are as follows:

Among them, state values are as follows:

updateReason values are as follows:

API Reference List