Custom Video Rendering

2024-05-21

Feature Overview

Custom video rendering refers to the SDK providing video frame data for local preview and remote stream playing to the outside for users to render themselves.

When the following situations occur in the developer's business, it is recommended to use the SDK's custom video rendering feature:

The app uses a cross-platform UI framework (e.g., Qt requires interfaces with complex hierarchical relationships to achieve high-experience interaction) or game engines (e.g., Unity, Unreal Engine, Cocos, etc.).
The app needs to obtain video frame data captured or played by the SDK for special processing.

Prerequisites

Before implementing the custom video rendering feature, please refer to the Writing Platform-Specific Code (Plugin Implementation) document to create a platform channel.

Usage Steps

The sequence diagram of API interface calls is as follows, where Native is iOS and Android:

Warning

Do not call the destroyEngine interface on the Native side (iOS or Android), otherwise it will cause functional abnormalities.
This document only explains how to enable custom video rendering on the Flutter side. For advanced features, please refer to iOS Custom Video Rendering and Android Custom Video Rendering documents.

1 Set Custom Video Rendering Configuration

Create a ZegoCustomVideoRenderConfig object and configure custom video rendering parameters. Call the enableCustomVideoRender interface to enable the custom video rendering feature.

ZegoCustomVideoRenderConfig config = ZegoCustomVideoRenderConfig(
    ZegoVideoBufferType.RawData,
    ZegoVideoFrameFormatSeries.RGB,
    false);
await ZegoExpressEngine.instance
    .enableCustomVideoRender(true, config);

ZegoCustomVideoRenderConfig config = ZegoCustomVideoRenderConfig(
    ZegoVideoBufferType.RawData,
    ZegoVideoFrameFormatSeries.RGB,
    false);
await ZegoExpressEngine.instance
    .enableCustomVideoRender(true, config);

2 Set Custom Video Rendering Callback

Add a setCustomVideoRenderHandler interface in the Flutter layer and call the Native layer through MethodChannel.

// Needs to be implemented by the developer
class ExpressTestImpl {
  final MethodChannel _channel =
      MethodChannel('plugins.zego.im/zego_express_test_demo');

  // Implement Flutter calling Native interface
  Future<void> setCustomVideoRenderHandler() async {
    await _channel.invokeMethod('setCustomVideoRenderHandler');
  }
}

// Needs to be implemented by the developer
class ExpressTestImpl {
  final MethodChannel _channel =
      MethodChannel('plugins.zego.im/zego_express_test_demo');

  // Implement Flutter calling Native interface
  Future<void> setCustomVideoRenderHandler() async {
    await _channel.invokeMethod('setCustomVideoRenderHandler');
  }
}

Implement the setCustomVideoRenderHandler interface capability in the Native layer.

Android Example

    // CustomVideoRender.java
    // Implement IZegoFlutterCustomVideoRenderHandler
    public class CustomVideoRender implements IZegoFlutterCustomVideoRenderHandler {
        @SuppressLint("StaticFieldLeak")
        private static volatile CustomVideoRender instance;

        private CustomVideoRender() {}

        public static CustomVideoRender getInstance() {
            if (instance == null) {
                synchronized (CustomVideoRender.class) {
                    if (instance == null) {
                        instance = new CustomVideoRender();
                    }
                }
            }
            return instance;
        }

        @Override
        public void onCapturedVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZGFlutterVideoFrameParam param, ZGFlutterVideoFlipMode flipMode, ZGFlutterPublishChannel channel) {
        }

        @Override
        public void onRemoteVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZGFlutterVideoFrameParam param, String streamID) {
        }

        @Override
        public void onRemoteVideoFrameEncodedData(ByteBuffer data, int dataLength, ZGFlutterVideoEncodedFrameParam param, long referenceTimeMillisecond, String streamID) {
        }
    }

    // CustomVideoRender.java
    // Implement IZegoFlutterCustomVideoRenderHandler
    public class CustomVideoRender implements IZegoFlutterCustomVideoRenderHandler {
        @SuppressLint("StaticFieldLeak")
        private static volatile CustomVideoRender instance;

        private CustomVideoRender() {}

        public static CustomVideoRender getInstance() {
            if (instance == null) {
                synchronized (CustomVideoRender.class) {
                    if (instance == null) {
                        instance = new CustomVideoRender();
                    }
                }
            }
            return instance;
        }

        @Override
        public void onCapturedVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZGFlutterVideoFrameParam param, ZGFlutterVideoFlipMode flipMode, ZGFlutterPublishChannel channel) {
        }

        @Override
        public void onRemoteVideoFrameRawData(ByteBuffer[] data, int[] dataLength, ZGFlutterVideoFrameParam param, String streamID) {
        }

        @Override
        public void onRemoteVideoFrameEncodedData(ByteBuffer data, int dataLength, ZGFlutterVideoEncodedFrameParam param, long referenceTimeMillisecond, String streamID) {
        }
    }

// ExpressTestPlugin.java
// methodChannel example
public class ExpressTestPlugin implements FlutterPlugin, MethodChannel.MethodCallHandler {

    private MethodChannel methodChannel;

    @Override
    public void onAttachedToEngine(@NonNull FlutterPluginBinding binding) {
        methodChannel = new MethodChannel(binding.getBinaryMessenger(), "plugins.zego.im/zego_express_test_demo");
        methodChannel.setMethodCallHandler(this);
    }

    @Override
    public void onDetachedFromEngine(@NonNull FlutterPluginBinding binding) {
        methodChannel.setMethodCallHandler(null);
    }

    @Override
    public void onMethodCall(@NonNull MethodCall call, @NonNull MethodChannel.Result result) {
        switch (call.method) {
            case "setCustomVideoRenderHandler": {
                ZegoCustomVideoRenderManager.getInstance().setCustomVideoRenderHandler(CustomVideoRender.getInstance());
                result.success(true);
                break;
            }
        }
    }
}

// ExpressTestPlugin.java
// methodChannel example
public class ExpressTestPlugin implements FlutterPlugin, MethodChannel.MethodCallHandler {

    private MethodChannel methodChannel;

    @Override
    public void onAttachedToEngine(@NonNull FlutterPluginBinding binding) {
        methodChannel = new MethodChannel(binding.getBinaryMessenger(), "plugins.zego.im/zego_express_test_demo");
        methodChannel.setMethodCallHandler(this);
    }

    @Override
    public void onDetachedFromEngine(@NonNull FlutterPluginBinding binding) {
        methodChannel.setMethodCallHandler(null);
    }

    @Override
    public void onMethodCall(@NonNull MethodCall call, @NonNull MethodChannel.Result result) {
        switch (call.method) {
            case "setCustomVideoRenderHandler": {
                ZegoCustomVideoRenderManager.getInstance().setCustomVideoRenderHandler(CustomVideoRender.getInstance());
                result.success(true);
                break;
            }
        }
    }
}

iOS Example

    // CustomVideoRender.h
    @interface CustomVideoRender : NSObject <ZegoFlutterCustomVideoRenderHandler>

    /// Get the custom video render manager instance
    + (instancetype)sharedInstance;

    @end

    // CustomVideoRender.h
    @interface CustomVideoRender : NSObject <ZegoFlutterCustomVideoRenderHandler>

    /// Get the custom video render manager instance
    + (instancetype)sharedInstance;

    @end

// CustomVideoRender.m
@interface CustomVideoRender()

@end

@implementation CustomVideoRender

+ (instancetype)sharedInstance {
    static CustomVideoRender *instance = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        instance = [[CustomVideoRender alloc] init];
    });
    return instance;
}

#pragma mark ZegoFlutterCustomVideoRenderHandler
- (void)onCapturedVideoFrameRawData:(unsigned char *_Nonnull *_Nonnull)data
                         dataLength:(unsigned int *)dataLength
                              param:(ZGFlutterVideoFrameParam *)param
                           flipMode:(ZGFlutterVideoFlipMode)flipMode
                            channel:(ZGFlutterPublishChannel)channel {
}

- (void)onRemoteVideoFrameRawData:(unsigned char *_Nonnull *_Nonnull)data
                       dataLength:(unsigned int *)dataLength
                            param:(ZGFlutterVideoFrameParam *)param
                         streamID:(NSString *)streamID {
}

- (void)onCapturedVideoFrameCVPixelBuffer:(CVPixelBufferRef)buffer
                                    param:(ZGFlutterVideoFrameParam *)param
                                 flipMode:(ZGFlutterVideoFlipMode)flipMode
                                  channel:(ZGFlutterPublishChannel)channel {
}

- (void)onRemoteVideoFrameCVPixelBuffer:(CVPixelBufferRef)buffer
                                  param:(ZGFlutterVideoFrameParam *)param
                               streamID:(NSString *)streamID {
}

- (void)onRemoteVideoFrameEncodedData:(unsigned char *_Nonnull)data
                           dataLength:(unsigned int)dataLength
                                param:(ZGFlutterVideoEncodedFrameParam *)param
             referenceTimeMillisecond:(unsigned long long)referenceTimeMillisecond
                             streamID:(NSString *)streamID {
}

@end

// CustomVideoRender.m
@interface CustomVideoRender()

@end

@implementation CustomVideoRender

+ (instancetype)sharedInstance {
    static CustomVideoRender *instance = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        instance = [[CustomVideoRender alloc] init];
    });
    return instance;
}

#pragma mark ZegoFlutterCustomVideoRenderHandler
- (void)onCapturedVideoFrameRawData:(unsigned char *_Nonnull *_Nonnull)data
                         dataLength:(unsigned int *)dataLength
                              param:(ZGFlutterVideoFrameParam *)param
                           flipMode:(ZGFlutterVideoFlipMode)flipMode
                            channel:(ZGFlutterPublishChannel)channel {
}

- (void)onRemoteVideoFrameRawData:(unsigned char *_Nonnull *_Nonnull)data
                       dataLength:(unsigned int *)dataLength
                            param:(ZGFlutterVideoFrameParam *)param
                         streamID:(NSString *)streamID {
}

- (void)onCapturedVideoFrameCVPixelBuffer:(CVPixelBufferRef)buffer
                                    param:(ZGFlutterVideoFrameParam *)param
                                 flipMode:(ZGFlutterVideoFlipMode)flipMode
                                  channel:(ZGFlutterPublishChannel)channel {
}

- (void)onRemoteVideoFrameCVPixelBuffer:(CVPixelBufferRef)buffer
                                  param:(ZGFlutterVideoFrameParam *)param
                               streamID:(NSString *)streamID {
}

- (void)onRemoteVideoFrameEncodedData:(unsigned char *_Nonnull)data
                           dataLength:(unsigned int)dataLength
                                param:(ZGFlutterVideoEncodedFrameParam *)param
             referenceTimeMillisecond:(unsigned long long)referenceTimeMillisecond
                             streamID:(NSString *)streamID {
}

@end

// ZegoExpressTestPlugin.h
// methodChannel example
@interface ZegoExpressTestPlugin : NSObject<FlutterPlugin>

@end

// ZegoExpressTestPlugin.h
// methodChannel example
@interface ZegoExpressTestPlugin : NSObject<FlutterPlugin>

@end

// ZegoExpressTestPlugin.m
// methodChannel example
@interface ZegoExpressTestPlugin ()

@property (nonatomic, weak) id<FlutterPluginRegistrar> registrar;

@property (nonatomic, strong) FlutterMethodChannel *methodChannel;

@end

@implementation ZegoExpressTestPlugin

+ (void)registerWithRegistrar:(NSObject<FlutterPluginRegistrar>*)registrar {
    ZegoExpressTestPlugin *instance = [[ZegoExpressTestPlugin alloc] init];

    instance.registrar = registrar;

    FlutterMethodChannel *methodChannel = [FlutterMethodChannel
      methodChannelWithName:@"plugins.zego.im/zego_express_test_demo"
            binaryMessenger:[registrar messenger]];
    [registrar addMethodCallDelegate:instance channel:methodChannel];
    instance.methodChannel = methodChannel;
}

- (void)detachFromEngineForRegistrar:(NSObject<FlutterPluginRegistrar>*)registrar {
    [_methodChannel setMethodCallHandler:nil];
    _methodChannel = nil;
    _registrar = nil;
}

#pragma mark - Handle Method Call

- (void)handleMethodCall:(FlutterMethodCall*)call result:(FlutterResult)result {
    SEL selector = NSSelectorFromString([NSString stringWithFormat:@"%@:result:", call.method]);

    // Handle unrecognized method
    if (![self respondsToSelector:selector]) {
        result(@(false));
        return;
    }

    NSMethodSignature *signature = [self methodSignatureForSelector:selector];
    NSInvocation* invocation = [NSInvocation invocationWithMethodSignature:signature];

    invocation.target = self;
    invocation.selector = selector;
    [invocation setArgument:&call atIndex:2];
    [invocation setArgument:&result atIndex:3];
    [invocation invoke];
}

- (void)setCustomVideoRenderHandler:(FlutterMethodCall*)call result:(FlutterResult)result {
    [[ZegoCustomVideoRenderManager sharedInstance] setCustomVideoRenderHandler:[CustomVideoRender sharedInstance]];
    result(@(true));
}

@end

// ZegoExpressTestPlugin.m
// methodChannel example
@interface ZegoExpressTestPlugin ()

@property (nonatomic, weak) id<FlutterPluginRegistrar> registrar;

@property (nonatomic, strong) FlutterMethodChannel *methodChannel;

@end

@implementation ZegoExpressTestPlugin

+ (void)registerWithRegistrar:(NSObject<FlutterPluginRegistrar>*)registrar {
    ZegoExpressTestPlugin *instance = [[ZegoExpressTestPlugin alloc] init];

    instance.registrar = registrar;

    FlutterMethodChannel *methodChannel = [FlutterMethodChannel
      methodChannelWithName:@"plugins.zego.im/zego_express_test_demo"
            binaryMessenger:[registrar messenger]];
    [registrar addMethodCallDelegate:instance channel:methodChannel];
    instance.methodChannel = methodChannel;
}

- (void)detachFromEngineForRegistrar:(NSObject<FlutterPluginRegistrar>*)registrar {
    [_methodChannel setMethodCallHandler:nil];
    _methodChannel = nil;
    _registrar = nil;
}

#pragma mark - Handle Method Call

- (void)handleMethodCall:(FlutterMethodCall*)call result:(FlutterResult)result {
    SEL selector = NSSelectorFromString([NSString stringWithFormat:@"%@:result:", call.method]);

    // Handle unrecognized method
    if (![self respondsToSelector:selector]) {
        result(@(false));
        return;
    }

    NSMethodSignature *signature = [self methodSignatureForSelector:selector];
    NSInvocation* invocation = [NSInvocation invocationWithMethodSignature:signature];

    invocation.target = self;
    invocation.selector = selector;
    [invocation setArgument:&call atIndex:2];
    [invocation setArgument:&result atIndex:3];
    [invocation invoke];
}

- (void)setCustomVideoRenderHandler:(FlutterMethodCall*)call result:(FlutterResult)result {
    [[ZegoCustomVideoRenderManager sharedInstance] setCustomVideoRenderHandler:[CustomVideoRender sharedInstance]];
    result(@(true));
}

@end

Call ExpressTestImpl.setCustomVideoRenderHandler to set the custom video rendering callback.

ExpressTestImpl.instance.setCustomVideoRenderHandler();

ExpressTestImpl.instance.setCustomVideoRenderHandler();

3 Subsequent Operations

This document only explains how to enable custom video rendering on the Flutter side. For advanced features, please refer to iOS Custom Video Rendering and Android Custom Video Rendering documents.