Custom Video Rendering

2026-03-05

Feature Overview

Custom video rendering means that the SDK provides video frame data for local preview and remote play streams to the outside for users to render themselves.

When the following situations occur in developers' business, it is recommended to use the SDK's custom video rendering feature:

The App uses cross-platform UI frameworks (e.g., Qt requires interfaces with complex hierarchical relationships to achieve high-experience interactions) or game engines (e.g., Unity, Unreal Engine, Cocos, etc.).
The App needs to obtain video frame data captured or played by the SDK for special processing.

Example Source Code Download

Please refer to Download Example Source Code to get the source code.

For related source code, please check the files in the "/ZegoExpressExample/AdvancedVideoProcessing/src/main/java/im/zego/advancedvideoprocessing/customrender" directory.

Prerequisites

Before implementing custom video rendering functionality, please ensure:

You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
You have integrated ZEGO Express SDK in your project and implemented basic audio and video streaming functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

The API interface call sequence diagram is as follows:

1 Set up custom video rendering configuration

Create a ZegoCustomVideoRenderConfig object to configure custom video rendering parameters, where:
- bufferType, refers to the custom video rendering video frame data type. For values, please refer to ZegoVideoBufferType.
- frameFormatSeries, refers to the custom video rendering video frame data format. For values, please refer to ZegoVideoFrameFormatSeries. This parameter can only specify the RGB or YUV color space category. The specific data format varies between platforms and is subject to the parameters in the callback.
  
  Warning
  If the data format in the callback does not match your expectations, please contact ZEGO Technical Support for assistance.
  - ZegoVideoFrameFormatSeriesRGB: Returns RGBA.
  - ZegoVideoFrameFormatSeriesYUV: Returns I420.
- enableEngineRender, refers to whether the SDK also renders internally while custom video rendering is being performed. The default is "false", which means the Engine will not render on the View set by the preview interface startPreview and the play stream interface startPlayingStream.

Call the enableCustomVideoRender interface to enable the custom video rendering feature.

ZegoCustomVideoRenderConfig videoRenderConfig = new ZegoCustomVideoRenderConfig();
// Custom video rendering video frame data type, taking RAW_DATA type as an example
videoRenderConfig.bufferType = ZegoVideoBufferType.RAW_DATA;
// Custom video rendering video frame data format, taking RGB color series data format as an example
videoRenderConfig.frameFormatSeries = ZegoVideoFrameFormatSeries.RGB;
// Specify that the Engine also renders while custom video rendering is being performed
videoRenderConfig.enableEngineRender = true;

// Enable custom video rendering
engine.enableCustomVideoRender(true, videoRenderConfig);

ZegoCustomVideoRenderConfig videoRenderConfig = new ZegoCustomVideoRenderConfig();
// Custom video rendering video frame data type, taking RAW_DATA type as an example
videoRenderConfig.bufferType = ZegoVideoBufferType.RAW_DATA;
// Custom video rendering video frame data format, taking RGB color series data format as an example
videoRenderConfig.frameFormatSeries = ZegoVideoFrameFormatSeries.RGB;
// Specify that the Engine also renders while custom video rendering is being performed
videoRenderConfig.enableEngineRender = true;

// Enable custom video rendering
engine.enableCustomVideoRender(true, videoRenderConfig);

2 Set up video renderer object and implement callback methods

Call the setCustomVideoRenderHandler interface to set up custom video rendering callbacks.

// Set up custom video rendering callback and implement the method for throwing out video data

engine.setCustomVideoRenderHandler(new IZegoCustomVideoRenderHandler(){

    public void onCapturedVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZegoVideoFrameParam param, ZegoVideoFlipMode flipMode, ZegoPublishChannel channel){
        // In the capture end callback, implement the logic of rendering the locally captured video data to the View through parameters such as data, dataLength, param, etc. The thrown data format refers to param.format
        ...;
    }

    public void onRemoteVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZegoVideoFrameParam param, String streamID){
        // In the play stream end callback, implement the logic of rendering the video data of the played stream to the View through parameters such as data, dataLength, param, etc. The thrown data format refers to param.format
        ...;
    }
});

// Set up custom video rendering callback and implement the method for throwing out video data

engine.setCustomVideoRenderHandler(new IZegoCustomVideoRenderHandler(){

    public void onCapturedVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZegoVideoFrameParam param, ZegoVideoFlipMode flipMode, ZegoPublishChannel channel){
        // In the capture end callback, implement the logic of rendering the locally captured video data to the View through parameters such as data, dataLength, param, etc. The thrown data format refers to param.format
        ...;
    }

    public void onRemoteVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZegoVideoFrameParam param, String streamID){
        // In the play stream end callback, implement the logic of rendering the video data of the played stream to the View through parameters such as data, dataLength, param, etc. The thrown data format refers to param.format
        ...;
    }
});

Note

The flipMode parameter in the local preview capture video frame callback method is related to mirroring. It notifies developers whether they need to flip the video frame image themselves to make the image conform to the description of the ZegoVideoMirrorMode enum value set in setVideoMirrorMode.

The param parameter (ZegoVideoFrameParam object) in the above callback methods describes some parameters of the video frame, defined as follows:

/**
 * Video frame parameter object
 *
 * Includes the format, width, height, etc. of the video frame
 */
public class ZegoVideoFrameParam {

    // Format of the video frame
    public ZegoVideoFrameFormat format;

    // Number of bytes per line for each plane (this parameter is an int array with a length of 4, RGBA only needs to consider strides[0], I420 needs to consider strides[0,1,2])
    final public int[] strides = new int[4];

    // Image width of the video frame
    public int width;

    // Image height of the video frame
    public int height;

}

/**
 * Video frame parameter object
 *
 * Includes the format, width, height, etc. of the video frame
 */
public class ZegoVideoFrameParam {

    // Format of the video frame
    public ZegoVideoFrameFormat format;

    // Number of bytes per line for each plane (this parameter is an int array with a length of 4, RGBA only needs to consider strides[0], I420 needs to consider strides[0,1,2])
    final public int[] strides = new int[4];

    // Image width of the video frame
    public int width;

    // Image height of the video frame
    public int height;

}

Where format identifies the specific data format of the video frame, strides is an array describing the number of bytes per line for each plane, and size describes the image size of the video frame. The relationship between strides and the image is shown in the figure:

3 Custom video rendering video frame data callback

Publish stream preview rendering

The publishing end first needs to call the start preview interface to receive custom video rendering video frame data callbacks. If the enableEngineRender parameter of the ZegoCustomVideoRenderConfig custom video rendering configuration is false, the canvas parameter for starting preview can be passed as empty. After starting preview, you can start publishing stream.

// If you need the Engine to also render internally while custom video rendering is being performed, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `true`, then pass in the View for internal rendering when previewing
ZegoCanvas previewCanvas = new ZegoCanvas(textureViewLocalPreview);// textureViewLocalPreview is the TextureView object on the UI interface
ZegoExpressEngine.getEngine().startPreview(previewCanvas);

// If you only need custom video rendering, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `false`, and pass the `canvas` parameter as empty. However, you must still call this interface, otherwise custom video rendering will not callback preview video frame data
ZegoExpressEngine.getEngine().startPreview(null);

// After starting preview, at this point you will receive custom video rendering preview video frame data callbacks

// Start publishing stream
// streamid is the stream ID defined by the developer
ZegoExpressEngine.getEngine().startPublishingStream(streamid);

// If you need the Engine to also render internally while custom video rendering is being performed, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `true`, then pass in the View for internal rendering when previewing
ZegoCanvas previewCanvas = new ZegoCanvas(textureViewLocalPreview);// textureViewLocalPreview is the TextureView object on the UI interface
ZegoExpressEngine.getEngine().startPreview(previewCanvas);

// If you only need custom video rendering, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `false`, and pass the `canvas` parameter as empty. However, you must still call this interface, otherwise custom video rendering will not callback preview video frame data
ZegoExpressEngine.getEngine().startPreview(null);

// After starting preview, at this point you will receive custom video rendering preview video frame data callbacks

// Start publishing stream
// streamid is the stream ID defined by the developer
ZegoExpressEngine.getEngine().startPublishingStream(streamid);

Play stream rendering

After successful publishing, you can call enableEngineRender in ZegoCustomVideoRenderConfig to set play stream rendering parameters. Then call startPlayingStream to play stream.

// If you need the Engine to also render internally while custom video rendering is being performed, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `true`, then pass in the View for internal rendering when playing stream
// textureViewLocalPreview is the TextureView object on the UI interface
ZegoCanvas playCanvas = new ZegoCanvas(textureViewLocalPreview);
ZegoExpressEngine.getEngine().startPlayingStream(streamID, playCanvas);

// If you only need custom video rendering, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `false`, and pass the `canvas` parameter as empty
egoExpressEngine.getEngine().startPlayingStream(streamID, null);

// After starting to play stream, at this point you will receive custom video rendering video frame data callbacks for this played stream

// If you need the Engine to also render internally while custom video rendering is being performed, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `true`, then pass in the View for internal rendering when playing stream
// textureViewLocalPreview is the TextureView object on the UI interface
ZegoCanvas playCanvas = new ZegoCanvas(textureViewLocalPreview);
ZegoExpressEngine.getEngine().startPlayingStream(streamID, playCanvas);

// If you only need custom video rendering, you can set the `enableEngineRender` parameter of `ZegoCustomVideoRenderConfig` to `false`, and pass the `canvas` parameter as empty
egoExpressEngine.getEngine().startPlayingStream(streamID, null);

// After starting to play stream, at this point you will receive custom video rendering video frame data callbacks for this played stream

At this point, the App successfully obtains the video frame data callbacked by the SDK for actual rendering actions or deep processing operations.

FAQ

Custom video rendering is divided into local capture preview custom video rendering and remote play stream custom video rendering. What are their respective roles? In what scenarios are they used?

Local capture preview custom video rendering allows the host to see additional rendering effects, adding special effects etc. on top of the original video image; remote play stream custom video rendering allows the audience to see different rendering images, and special effects can be added on top of the played stream image according to the audience's preferences.
If custom video rendering is configured with ZegoCustomVideoRenderConfig and the enableEngineRender parameter is false, what should be filled in for the canvas parameter in the preview interface startPreview and play stream interface startPlayingStream?

When enableEngineRender is false, the Engine does not render, so the canvas parameter for preview and play stream interfaces can be set to "null".
When using custom video rendering during publishing, will the processed preview video data buffer only be displayed locally? Will it be included in the published stream?

It will only be displayed locally and does not affect the video data published out.
What are the width and height of the video frames for local capture preview custom video rendering?

The width and height of the video frames for local capture preview custom video rendering will be returned in the param parameter in the callback. Their values are the same as the resolution set by setVideoConfig.
What is the data format of the video frames output by custom video rendering? Is YUV supported?

It is determined by the selected rendering data format series ZegoVideoFrameFormatSeries and the data directly returned after soft decoding/hard decoding, that is, the format (ZegoVideoFrameFormat) parameter in the param parameter of the video frame callback method.

Video frame data format ZegoVideoFrameFormat description:

Video frame data format enum value	Description
I420	YUV420P, one YUV 12 bits group, Y, U, V three planes, four Y share one UV group.
NV12	YUV420SP, one YUV 12 bits group, Y, UV two planes, UV plane data arranged in U then V order, four Y share one UV group.
NV21	YUV420SP, one YUV 12 bits group, Y, UV two planes, UV plane data arranged in V then U order, four Y share one UV group.
BGRA32	BGRA32.
RGBA32	RGBA32.
ARGB32	ARGB32.
ABGR32	ABGR32.
I422	YUV422P, one YUV 16 bits group, two Y share one UV group.
BGR24	BGR24.
RGB24	RGB24.

What is the callback frequency per second for custom video rendering? What should be noted?

The callback frequency for local capture preview custom video rendering is generally the same as the frame rate set during publishing. However, if flow control is enabled and the control attributes include frame rate, the callback frequency for local capture preview custom video rendering will change accordingly; the callback frequency for remote play stream custom video rendering will also change with the received video data frame rate, such as the publishing end enabling flow control causing frame rate changes, play stream end network stuttering, play stream end network recovery and the SDK starting to catch up frames, all will affect the callback frequency for play stream custom video rendering.
How to get the first frame data in custom video rendering?

The data returned for the first time by the custom video rendering callback IZegoCustomVideoRenderHandler is the first frame data.
Is it supported to do local capture preview rendering ourselves and let ZEGO SDK do the play stream rendering?

Yes. When configuring ZegoCustomVideoRenderConfig for custom video rendering, setting the enableEngineRender parameter to "true" means that not only will the custom video renderer callback video frame data, but the SDK will also perform internal rendering on the View in the canvas of the preview interface startPreview and play stream interface startPlayingStream.
When rendering the preview view, why can't I receive the video data callback?

If you need to render the preview view, you need to start preview startPreview before publishing, otherwise you will not receive the video data callback.
For local preview custom video rendering, why is the video frame data image from the phone's front camera not horizontally mirrored by default?

The video frame image flipping for custom video rendering needs to be implemented by the developer. You can know whether the frame needs to be flipped through the flipMode in the video frame data callback interface.