Custom Video Rendering

2024-05-21

Feature Overview

Custom video rendering refers to the SDK providing local preview and remote stream playing video frame data to the outside for users to render themselves.

When the following situations occur in the developer's business, we recommend using the SDK's custom video rendering feature:

The App uses cross-platform UI frameworks (for example, QT requires interfaces with complex hierarchical relationships to achieve high-experience interaction) or game engines (for example, Unity, Unreal Engine, Cocos, etc.)
The App needs to obtain video frame data captured or played by the SDK for special processing

Download Sample Source Code

Please refer to Download Sample Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/Examples/AdvancedVideoProcessing/CustomVideoRendering" directory.

Implementation Steps

The usage flow of custom video rendering is as follows:

The sequence diagram is as follows:

1 Set custom video rendering configuration

Create ZegoCustomVideoRenderConfig object and configure parameters.

The bufferType parameter is the enumeration ZegoVideoBufferType . Currently, the SDK only supports the ZEGO_VIDEO_BUFFER_TYPE_RAWDATA type.

The frameFormatSeries parameter is the enumeration ZegoVideoFrameFormatSeries , which can specify the custom video rendering video frame data format required by the developer. This parameter can only specify the RGB or YUV color space categories. The specific data format varies between platforms and is subject to the parameters in the callback, which will be mentioned in the second half of this article.

Warning

If the data format in the callback does not match your expectations, please contact ZEGOCLOUD Technical Support for handling.

ZegoVideoFrameFormatSeriesRGB: Returns BGRA.
ZegoVideoFrameFormatSeriesYUV: Returns I420.

enableEngineRender indicates whether the SDK also renders internally while custom video rendering is being performed. If it is false, the engine will not render on the canvas set by the preview interface startPreview and the playing stream interface startPlayingStream .

Interface prototype

// Custom video rendering configuration
struct ZegoCustomVideoRenderConfig
{
    // Custom video rendering video frame data type
    ZegoVideoBufferType bufferType;

    // Custom video rendering video frame data format
    ZegoVideoFrameFormatSeries frameFormatSeries;

    // Whether the engine also renders during custom video rendering, default is [false]
    bool enableEngineRender;
};

/**
 * Start or stop custom video rendering
 *
 * Must be set before the engine starts, i.e., before calling [startPreview], [startPublishing], [startPlayingStream]; and the configuration can only be modified after the engine stops
 * When the developer enables custom rendering, by calling [setCustomVideoRenderHandler], you can set to receive local and remote video frame data for custom rendering
 *
 * @param enable Whether to enable
 * @param config Custom rendering configuration
 */
virtual void enableCustomVideoRender(bool enable, ZegoCustomVideoRenderConfig* config) = 0;

// Custom video rendering configuration
struct ZegoCustomVideoRenderConfig
{
    // Custom video rendering video frame data type
    ZegoVideoBufferType bufferType;

    // Custom video rendering video frame data format
    ZegoVideoFrameFormatSeries frameFormatSeries;

    // Whether the engine also renders during custom video rendering, default is [false]
    bool enableEngineRender;
};

/**
 * Start or stop custom video rendering
 *
 * Must be set before the engine starts, i.e., before calling [startPreview], [startPublishing], [startPlayingStream]; and the configuration can only be modified after the engine stops
 * When the developer enables custom rendering, by calling [setCustomVideoRenderHandler], you can set to receive local and remote video frame data for custom rendering
 *
 * @param enable Whether to enable
 * @param config Custom rendering configuration
 */
virtual void enableCustomVideoRender(bool enable, ZegoCustomVideoRenderConfig* config) = 0;

Call the enableCustomVideoRender interface to enable custom video rendering.

After creating the ZegoCustomVideoRenderConfig object and configuring parameters, call the enableCustomVideoRender interface to enable custom video rendering.

Call example

ZegoCustomVideoRenderConfig renderConfig;
renderConfig.bufferType = ZEGO_VIDEO_BUFFER_TYPE_RAWDATA;
renderConfig.frameFormatSeries = ZEGO_VIDEO_FRAME_FORMAT_SERIES_RGB;
renderConfig.enableEngineRender = false;

engine->enableCustomVideoRender(true, &renderConfig);

ZegoCustomVideoRenderConfig renderConfig;
renderConfig.bufferType = ZEGO_VIDEO_BUFFER_TYPE_RAWDATA;
renderConfig.frameFormatSeries = ZEGO_VIDEO_FRAME_FORMAT_SERIES_RGB;
renderConfig.enableEngineRender = false;

engine->enableCustomVideoRender(true, &renderConfig);

2 Set custom video rendering callback

Call the setCustomVideoRenderHandler interface to set the custom video rendering callback IZegoCustomVideoRenderHandler .

// Inherit from IZegoCustomVideoRenderHandler, implement custom rendering callback
class MyCustomVideoRenderHandler: public IZegoCustomVideoRenderHandler{
    /**
    * Local preview video frame raw data callback
    *
    * @param data Raw data of the video frame (e.g., for RGBA, only consider data[0]; for I420, consider data[0,1,2])
    * @param dataLength Length of the data (e.g., for RGBA, only consider dataLength[0]; for I420, consider dataLength[0,1,2])
    * @param param Video frame parameters
    * @param flipMode Video frame flip mode
    * @param channel Publishing channel
    */
    virtual void onCapturedVideoFrameRawData(unsigned char** /*data*/, unsigned int* /*dataLength*/, ZegoVideoFrameParam /*param*/, ZegoVideoFlipMode /*flipMode*/, ZegoPublishChannel /*channel*/) {

    }

    /**
    * Remote stream playing video frame raw data callback, distinguished by streamID
    *
    * @param data Raw data of the video frame (e.g., for RGBA, only consider data[0]; for I420, consider data[0,1,2])
    * @param dataLength Length of the data (e.g., for RGBA, only consider dataLength[0]; for I420, consider dataLength[0,1,2])
    * @param param Video frame parameters
    * @param streamID Stream ID of the played stream
    */
    virtual void onRemoteVideoFrameRawData(unsigned char** /*data*/, unsigned int* /*dataLength*/, ZegoVideoFrameParam /*param*/, const std::string& /*streamID*/) {

    }
}

// Set custom rendering callback for engine
engine->setCustomVideoRenderHandler(std::make_shared<MyCustomVideoRenderHandler>());

// Inherit from IZegoCustomVideoRenderHandler, implement custom rendering callback
class MyCustomVideoRenderHandler: public IZegoCustomVideoRenderHandler{
    /**
    * Local preview video frame raw data callback
    *
    * @param data Raw data of the video frame (e.g., for RGBA, only consider data[0]; for I420, consider data[0,1,2])
    * @param dataLength Length of the data (e.g., for RGBA, only consider dataLength[0]; for I420, consider dataLength[0,1,2])
    * @param param Video frame parameters
    * @param flipMode Video frame flip mode
    * @param channel Publishing channel
    */
    virtual void onCapturedVideoFrameRawData(unsigned char** /*data*/, unsigned int* /*dataLength*/, ZegoVideoFrameParam /*param*/, ZegoVideoFlipMode /*flipMode*/, ZegoPublishChannel /*channel*/) {

    }

    /**
    * Remote stream playing video frame raw data callback, distinguished by streamID
    *
    * @param data Raw data of the video frame (e.g., for RGBA, only consider data[0]; for I420, consider data[0,1,2])
    * @param dataLength Length of the data (e.g., for RGBA, only consider dataLength[0]; for I420, consider dataLength[0,1,2])
    * @param param Video frame parameters
    * @param streamID Stream ID of the played stream
    */
    virtual void onRemoteVideoFrameRawData(unsigned char** /*data*/, unsigned int* /*dataLength*/, ZegoVideoFrameParam /*param*/, const std::string& /*streamID*/) {

    }
}

// Set custom rendering callback for engine
engine->setCustomVideoRenderHandler(std::make_shared<MyCustomVideoRenderHandler>());

The param (ZegoVideoFrameParam object) in the callback method describes some parameters of the video frame, defined as follows:

struct ZegoVideoFrameParam
{
    // Format of the video frame
    ZegoVideoFrameFormat format;

    // Number of bytes per row for each plane (e.g., for RGBA, only consider strides[0]; for I420, consider strides[0,1,2])
    int strides[4];

    // Width of the video frame
    int width;

    // Height of the video frame
    int height;

    // Video rotation angle
    int rotation;
};

struct ZegoVideoFrameParam
{
    // Format of the video frame
    ZegoVideoFrameFormat format;

    // Number of bytes per row for each plane (e.g., for RGBA, only consider strides[0]; for I420, consider strides[0,1,2])
    int strides[4];

    // Width of the video frame
    int width;

    // Height of the video frame
    int height;

    // Video rotation angle
    int rotation;
};

Among them, format identifies the specific data format of the video frame; strides is an array describing the number of bytes per row for each plane; size describes the screen size of the video frame.

The relationship between strides and the image is shown in the figure:

After enabling local preview or publishing stream, the SDK will callback the captured local video to the user through onCapturedVideoFrameRawData .

After enabling playing stream, the SDK will callback the obtained remote stream video data to the user through onRemoteVideoFrameRawData .

At this point, the App successfully obtains the video frame data callbacked by the SDK for actual rendering actions or deep processing operations.

FAQ

Custom video rendering is divided into local capture preview custom video rendering and remote stream playing custom video rendering. What are their respective roles? In what scenarios are they used?

Answer: Local capture preview custom video rendering allows the host to see additional rendering effects, adding effects etc. on top of the original video screen; remote stream playing custom video rendering allows the audience to see different rendering screens, and effects can be added on top of the playing screen according to the audience's preferences.
If custom video rendering sets the enableEngineRender parameter in the configuration ZegoCustomVideoRenderConfig to false, what should be filled in for the canvas parameter in the preview interface startPreview and the playing stream interface startPlayingStream ?

Answer: When enableEngineRender is false, the engine does not render, so the canvas parameter of the preview and playing stream interfaces can be set to null nullptr.
When using custom video rendering during publishing stream, will the processed preview video data buffer only be displayed locally? Will it be included in the published stream?

Answer: It will only be displayed locally and will not affect the video data published out.
What is the width and height of the video frame for local capture preview custom video rendering?

Answer: The width and height of the video frame for local capture preview custom video rendering will be returned in the param parameter in the callback, and its value is the same as the resolution set by setVideoConfig .
What is the video frame data format output by custom video rendering?

Answer: It is determined by the selected rendering data format series ZegoVideoFrameFormatSeries and the data directly returned after soft decoding/hard decoding, i.e., the format parameter in the ZegoVideoFrameParam parameter in the video frame callback method.

Video frame data format ZegoVideoFrameFormat description:

Video frame data format enumeration value	Description
ZEGO_VIDEO_FRAME_FORMAT_I420	YUV420P; one YUV 12bits; Y, U, V three planes, four Y share one set of UV.
ZEGO_VIDEO_FRAME_FORMAT_NV12	YUV420SP; one YUV 12bits; Y, UV two planes, UV plane data arranged in U first V second order, four Y share one set of UV.
ZEGO_VIDEO_FRAME_FORMAT_NV21	YUV420SP; one YUV 12bits; Y, UV two planes, UV plane data arranged in V first U second order, four Y share one set of UV.
ZEGO_VIDEO_FRAME_FORMAT_BGRA32	BGRA32.
ZEGO_VIDEO_FRAME_FORMAT_RGBA32	RGBA32.
ZEGO_VIDEO_FRAME_FORMAT_ARGB32	ARGB32.
ZEGO_VIDEO_FRAME_FORMAT_ABGR32	ABGR32.
ZEGO_VIDEO_FRAME_FORMAT_I422	YUV422P; one YUV 16bits; two Y share one set of UV.
ZEGO_VIDEO_FRAME_FORMAT_BGR24	BGR24.
ZEGO_VIDEO_FRAME_FORMAT_RGB24	RGB24.

What is the callback frequency per second for custom video rendering? What should be noted?

Answer: The callback frequency of local capture preview custom video rendering is generally the same as the frame rate set during publishing. However, if flow control is enabled and the control attributes include frame rate, the callback frequency of local capture preview custom video rendering will change accordingly; the callback frequency of remote stream playing custom video rendering will also change with the frame rate of the received video data, such as the publishing side enabling flow control causing frame rate changes, the playing side network stuttering, or the playing side network recovering and the SDK starting to catch up frames, all of which will affect the callback frequency of playing stream custom video rendering.
How to get the first frame data in custom video rendering?

Answer: The data returned for the first time in the custom video rendering callback IZegoCustomVideoRenderHandler is the first frame data.
Why can't I receive the local video data callback?

Answer: You need to start preview startPreview before publishing stream, otherwise you won't receive the video data callback.
Why doesn't the local preview custom video rendering video frame screen perform horizontal mirror flipping by default?

Answer: The video frame screen flipping for custom video rendering needs to be implemented by the developer himself. You can know whether the frame needs to be flipped through the flipMode in the video frame data callback interface.