How to implement screen sharing through custom capture?

Feature Introduction

Screen sharing refers to sharing screen content as video to other viewers during video calls or live streaming to enhance interactive experience and improve communication efficiency.

Screen sharing is widely used in the following scenarios:

• In video conference scenarios, screen sharing can share the speaker's local files, data, web pages, PPT and other content to other participants;
• In online classroom scenarios, screen sharing can display the teacher's courseware, notes, lecture content and other content to students.

iOS Implementation Process

Prerequisites

Before implementing screen sharing functionality, please ensure:

• You have prepared an iOS device or simulator with iOS 12.0 or above that supports audio and video (real device is recommended).
• You have integrated ZEGO Express SDK in your project and implemented basic Video Call functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementing Video Call.
• You have created a project in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to "Project Information" in Console - Project Management.

Implementation Process

The iOS platform implements screen recording based on Apple's Replaykit framework, which can share the screen content of the entire system. However, it requires the current App (main App process) to additionally provide an Extension component (Extension process) for screen recording, combined with ZEGO Express SDK related APIs to implement screen sharing functionality.

The main process of implementing screen sharing is as follows:

1. In your project, create a new Broadcast Upload Extension Target and integrate ZEGO Express SDK in it.
2. Create App Groups in the Broadcast Upload Extension Target so that the Extension process can synchronize configuration data with the main App process.
3. Use the Extension process to record the screen and send the recorded screen buffer data to ZEGO Express SDK through handleReplayKitSampleBuffer.
4. Manually trigger screen sharing.

For detailed operations, please refer to the following.

Usage Steps

Create a new Broadcast Upload Extension in the project

1. Open the project file with Xcode, and click "File > New > Target..." in the menu bar.

2. In the pop-up window, select "Broadcast Upload Extension" on the iOS page and click "Next".

3. In the pop-up dialog, fill in the name of "Broadcast Upload Extension" in the "Product Name" field, for example "ScreenShare". After selecting "Team", "Language" and other information, click "Finish".

After creation, you will see the Extension folder in the project. The structure is similar to the following. This folder is used to store the implementation code for screen sharing functionality:

(Optional) Synchronize configuration data through App Groups

Create App Groups

1. In the newly created Target in Xcode, select "Signing & Capabilities > Capability > App Groups".

2. In the pop-up window, add "App Groups ID" to bind the Extension process with the main App process.

3. Use iOS system API NSUserDefaults to synchronize configuration data (such as AppID, AppSign, roomID, etc.) between the Extension process and the main App process.

NSUserDefaults *userDefaults = [[NSUserDefaults alloc] initWithSuiteName:appGroup];

NSUserDefaults *userDefaults = [[NSUserDefaults alloc] initWithSuiteName:appGroup];

Get screen data through Extension and publish stream

1. Ensure that in the Extension's "Info.plist" file, "RPBroadcastProcessMode" is set to "RPBroadcastProcessModeSampleBuffer".

2. Import ZEGO Express SDK in the Extension. For details, please refer to "Import SDK" in Quick Start - Integration.

3. The system notifies the Extension through broadcastStartedWithSetupInfo callback that screen recording has started. You need to call the following interfaces of ZEGO Express SDK in this callback to implement related functions:

   1. Create ZegoExpressEngine.

   ZegoEngineConfig *engineConfig = [[ZegoEngineConfig alloc] init];
   
   engineConfig.advancedConfig = @{
       @"replaykit_handle_rotation": @"false", // Specify not to process the screen rotation on the publisher side, but to process it on the player side, thereby reduce memory usage, but in this case, the player must not play this stream from the CDN but directly from the ZEGO server. If you need to play this stream from the CDN and the captured screen needs to be dynamically rotated, please comment this line of code.
       @"max_channels": @"0", // Specify the max number of streams to play as 0  (Because this extension only needs to publish stream)
       @"max_publish_channels": @"1" // Specify the max number of streams to publish as 1
   };
   
   // Set engine config
   [ZegoExpressEngine setEngineConfig:engineConfig];
   
   // Create engine
   ZegoEngineProfile *profile = [ZegoEngineProfile new];
   profile.appID = appID;
   profile.appSign = appSign;
   profile.scenario = ZegoScenarioDefault;
   
   [ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
   
   // Enable hardware encoder to reduce memory usage when publishing stream
   [[ZegoExpressEngine sharedEngine] enableHardwareEncoder:YES];

   ZegoEngineConfig *engineConfig = [[ZegoEngineConfig alloc] init];
   
   engineConfig.advancedConfig = @{
       @"replaykit_handle_rotation": @"false", // Specify not to process the screen rotation on the publisher side, but to process it on the player side, thereby reduce memory usage, but in this case, the player must not play this stream from the CDN but directly from the ZEGO server. If you need to play this stream from the CDN and the captured screen needs to be dynamically rotated, please comment this line of code.
       @"max_channels": @"0", // Specify the max number of streams to play as 0  (Because this extension only needs to publish stream)
       @"max_publish_channels": @"1" // Specify the max number of streams to publish as 1
   };
   
   // Set engine config
   [ZegoExpressEngine setEngineConfig:engineConfig];
   
   // Create engine
   ZegoEngineProfile *profile = [ZegoEngineProfile new];
   profile.appID = appID;
   profile.appSign = appSign;
   profile.scenario = ZegoScenarioDefault;
   
   [ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
   
   // Enable hardware encoder to reduce memory usage when publishing stream
   [[ZegoExpressEngine sharedEngine] enableHardwareEncoder:YES];

   2. Initialize Express ReplayKit module.

   // Init SDK ReplayKit module
   [[ZegoExpressEngine sharedEngine] prepareForReplayKit];

   // Init SDK ReplayKit module
   [[ZegoExpressEngine sharedEngine] prepareForReplayKit];

   3. Login room and push screen sharing stream

   // Login room
   [[ZegoExpressEngine sharedEngine] loginRoom:self.roomID user:[ZegoUser userWithUserID:self.userID userName:self.userName]];
   // Push screen sharing stream
   [[ZegoExpressEngine sharedEngine] startPublishingStream:self.streamID];

   // Login room
   [[ZegoExpressEngine sharedEngine] loginRoom:self.roomID user:[ZegoUser userWithUserID:self.userID userName:self.userName]];
   // Push screen sharing stream
   [[ZegoExpressEngine sharedEngine] startPublishingStream:self.streamID];

4. In the processSampleBuffer system callback, send the screen recording buffer data to ZEGO Express SDK through handleReplayKitSampleBuffer.

   [[ZegoExpressEngine sharedEngine] handleReplayKitSampleBuffer:sampleBuffer bufferType:sampleBufferType];

   [[ZegoExpressEngine sharedEngine] handleReplayKitSampleBuffer:sampleBuffer bufferType:sampleBufferType];

5. The system notifies the Extension through broadcastFinished callback that screen recording has ended. You can call the following interfaces of ZEGO Express SDK in this callback to stop screen sharing:

   // 1. Stop pushing screen sharing stream.
   [[ZegoExpressEngine sharedEngine] stopPublishingStream];
   // 2. Logout room
   [[ZegoExpressEngine sharedEngine] logoutRoom:self.roomID];
   // 3. Destroy ZegoExpressEngine
   [ZegoExpressEngine destroyEngine];

   // 1. Stop pushing screen sharing stream.
   [[ZegoExpressEngine sharedEngine] stopPublishingStream];
   // 2. Logout room
   [[ZegoExpressEngine sharedEngine] logoutRoom:self.roomID];
   // 3. Destroy ZegoExpressEngine
   [ZegoExpressEngine destroyEngine];

Start screen sharing

You can trigger it in the following two ways:

• Method 1 (Recommended):

You need to long-press the screen recording button in the iOS system control center and select the corresponding Extension to start recording.

• Method 2:

Watch remote screen sharing

After completing the above steps, other users can use the startPlayingStream interface to pull the screen sharing stream. For detailed operations, please refer to Quick Start.

// Similarly, users pulling and playing streams first need to initialize the SDK and login to the same room
...
...

// play stream and play, need to pass in the streamID used by the user who initiated screen sharing
[[ZegoExpressEngine sharedEngine] startPlayingStream:streamID canvas:[ZegoCanvas canvasWithView:self.playView]];

// Similarly, users pulling and playing streams first need to initialize the SDK and login to the same room
...
...

// play stream and play, need to pass in the streamID used by the user who initiated screen sharing
[[ZegoExpressEngine sharedEngine] startPlayingStream:streamID canvas:[ZegoCanvas canvasWithView:self.playView]];

Android Implementation Process

Prerequisites

Before implementing screen sharing functionality, please ensure:

• You have integrated ZEGO Express SDK in your project and implemented basic Video Call functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementing Video Call.
• You have created a project in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to "Project Information" in Console - Project Management.

Implementation Process

We need to combine Android system API and ZEGO Express SDK's custom video capture to implement screen sharing.

The following diagram shows the data flow of implementing screen sharing on the Android platform:

Get user screen recording authorization

Before recording the screen, you need to get user authorization. Different versions require different permissions:

• Android 4.4 and earlier versions must obtain root permissions to implement screen recording. Since most devices now have system versions higher than 4.4, this scenario is not described here.
• Android 5.0 and above versions can use the MediaProjection and MediaProjectionManager provided by the system for screen recording. This version does not require root permissions, but will pop up a window asking the user whether to allow the application to record the screen, requiring user authorization.
• Android 10.0 and above versions require the use of foreground services when using system API for screen recording. For details, please refer to Official Documentation.

public static MediaProjectionManager mMediaProjectionManager;
if (Build.VERSION.SDK_INT < 21) {
    Toast.makeText(ZGVideoCaptureOriginUI.this, getString(R.string.record_request), Toast.LENGTH_SHORT).show();
    finish();
} else {
    // Version 5.0 and above
    // Request screen recording permission and wait for user authorization
   mMediaProjectionManager = (MediaProjectionManager) getSystemService(MEDIA_PROJECTION_SERVICE);
   startActivityForResult(mMediaProjectionManager.createScreenCaptureIntent(), REQUEST_CODE);
    }

public static MediaProjectionManager mMediaProjectionManager;
if (Build.VERSION.SDK_INT < 21) {
    Toast.makeText(ZGVideoCaptureOriginUI.this, getString(R.string.record_request), Toast.LENGTH_SHORT).show();
    finish();
} else {
    // Version 5.0 and above
    // Request screen recording permission and wait for user authorization
   mMediaProjectionManager = (MediaProjectionManager) getSystemService(MEDIA_PROJECTION_SERVICE);
   startActivityForResult(mMediaProjectionManager.createScreenCaptureIntent(), REQUEST_CODE);
    }

Create MediaProjection instance

1. Add related configuration in AndroidManifest.xml.

<application>
 <activity android:name="im.zego.videocapture.ui.ZGVideoCaptureDemoUI" />
 <activity android:name="im.zego.videocapture.ui.ZGVideoCaptureOriginUI"></activity>
 <service android:name=".service.CaptureScreenService"
     android:enabled="true"
     android:foregroundServiceType="mediaProjection"/>
</application>

<application>
 <activity android:name="im.zego.videocapture.ui.ZGVideoCaptureDemoUI" />
 <activity android:name="im.zego.videocapture.ui.ZGVideoCaptureOriginUI"></activity>
 <service android:name=".service.CaptureScreenService"
     android:enabled="true"
     android:foregroundServiceType="mediaProjection"/>
</application>

2. Create MediaProjection instance after user authorization.

• For Android 10.0 and below versions, directly get MediaProjection after authorization succeeds
• For Android 10.0 and above versions, the creation of MediaProjection instance needs to be executed in the onStartCommand method of the foreground service.

@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
   @Override
   protected void onActivityResult(int requestCode, int resultCode, Intent data) {
       super.onActivityResult(requestCode, resultCode, data);
       if (requestCode == REQUEST_CODE && resultCode == RESULT_OK) {
           if(Build.VERSION.SDK_INT>=Build.VERSION_CODES.Q){
               //Target version 10.0 and above requires using foreground service and creating MediaProjection in the onStartCommand method of the foreground service
               service=new Intent(ZGVideoCaptureOriginUI.this, CaptureScreenService.class);
               service.putExtra("code",resultCode);
               service.putExtra("data",data);
               startForegroundService(service);
           }else {
               //Target version below 10.0 directly get MediaProjection
               mMediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, data);
           }
       }
   }

@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
   @Override
   protected void onActivityResult(int requestCode, int resultCode, Intent data) {
       super.onActivityResult(requestCode, resultCode, data);
       if (requestCode == REQUEST_CODE && resultCode == RESULT_OK) {
           if(Build.VERSION.SDK_INT>=Build.VERSION_CODES.Q){
               //Target version 10.0 and above requires using foreground service and creating MediaProjection in the onStartCommand method of the foreground service
               service=new Intent(ZGVideoCaptureOriginUI.this, CaptureScreenService.class);
               service.putExtra("code",resultCode);
               service.putExtra("data",data);
               startForegroundService(service);
           }else {
               //Target version below 10.0 directly get MediaProjection
               mMediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, data);
           }
       }
   }

Create a class that implements the Service interface and create a MediaProjection instance in onStartCommand.

@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
public class CaptureScreenService extends Service {
    ...

    @Override
    public int onStartCommand(Intent intent, int flags, int startId) {
        ···
        //Get MediaProjection here
        ZGVideoCaptureOriginUI.mMediaProjection = ZGVideoCaptureOriginUI.mMediaProjectionManager.getMediaProjection(mResultCode, Objects.requireNonNull(mResultData));
        return super.onStartCommand(intent, flags, startId);
    }
   ···
}

@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
public class CaptureScreenService extends Service {
    ...

    @Override
    public int onStartCommand(Intent intent, int flags, int startId) {
        ···
        //Get MediaProjection here
        ZGVideoCaptureOriginUI.mMediaProjection = ZGVideoCaptureOriginUI.mMediaProjectionManager.getMediaProjection(mResultCode, Objects.requireNonNull(mResultData));
        return super.onStartCommand(intent, flags, startId);
    }
   ···
}

Enable custom video capture functionality of ZegoExpress SDK

Call the enableCustomVideoCapture interface of ZegoExpress SDK to enable custom capture functionality. For details, please refer to Custom Video Capture.

//VideoCaptureScreen inherits IZegoCustomVideoCaptureHandler to listen to custom capture onStart and onStop callbacks
VideoCaptureScreen videoCapture = new VideoCaptureScreen(ZGVideoCaptureOriginUI.mMediaProjection, DEFAULT_VIDEO_WIDTH, DEFAULT_VIDEO_HEIGHT, mSDKEngine);
//Listen to custom capture start and stop callbacks
mSDKEngine.setCustomVideoCaptureHandler(videoCapture);
ZegoCustomVideoCaptureConfig videoCaptureConfig=new ZegoCustomVideoCaptureConfig();
//Use SurfaceTexture type for custom capture
videoCaptureConfig.bufferType=ZegoVideoBufferType.SURFACE_TEXTURE;
//Start custom capture
mSDKEngine.enableCustomVideoCapture(true, videoCaptureConfig, ZegoPublishChannel.MAIN);

//VideoCaptureScreen inherits IZegoCustomVideoCaptureHandler to listen to custom capture onStart and onStop callbacks
VideoCaptureScreen videoCapture = new VideoCaptureScreen(ZGVideoCaptureOriginUI.mMediaProjection, DEFAULT_VIDEO_WIDTH, DEFAULT_VIDEO_HEIGHT, mSDKEngine);
//Listen to custom capture start and stop callbacks
mSDKEngine.setCustomVideoCaptureHandler(videoCapture);
ZegoCustomVideoCaptureConfig videoCaptureConfig=new ZegoCustomVideoCaptureConfig();
//Use SurfaceTexture type for custom capture
videoCaptureConfig.bufferType=ZegoVideoBufferType.SURFACE_TEXTURE;
//Start custom capture
mSDKEngine.enableCustomVideoCapture(true, videoCaptureConfig, ZegoPublishChannel.MAIN);

Login room and start pushing stream

Call the loginRoom interface, pass in the room ID parameter "roomID" and user parameter "user" to login to the room.

Call the startPublishingStream interface, pass in the stream ID parameter "streamID" to send your audio and video stream to remote users.

/** Create user */
ZegoUser user = new ZegoUser("user1");

/** Start logging into room */
mSDKEngine.loginRoom("room1", user);
/** Start pushing stream */
mSDKEngine.startPublishingStream("stream1");

/** Create user */
ZegoUser user = new ZegoUser("user1");

/** Start logging into room */
mSDKEngine.loginRoom("room1", user);
/** Start pushing stream */
mSDKEngine.startPublishingStream("stream1");

Create VirtualDisplay and send screen data to ZEGO Express SDK

1. Create ZegoVideoCaptureCallback class that inherits IZegoCustomVideoCaptureHandler.

2. Create VideoCaptureScreen class that inherits ZegoVideoCaptureCallback.

After receiving the onStart callback, developers can create a VirtualDisplay instance through MediaProjection to get screen data and send it to ZEGO Express SDK.

3. Use the createVirtualDisplay system API to render the virtual display content to the Surface.

//ZegoVideoCaptureCallback inherits from IZegoCustomVideoCaptureHandler
class VideoCaptureScreen extends ZegoVideoCaptureCallback {
    @Override
    //After receiving the onStart callback, you can create VirtualDisplay through MediaProjection and send screen data to ZEGO SDK
    public void onStart(ZegoPublishChannel channel) {
        if (mZegoEngine != null && !mIsCapturing && mMediaProjection != null) {
            mIsCapturing = true;
            //Get SurfaceTexture through ZEGO API getCustomVideoCaptureSurfaceTexture. This interface uses the main channel for pushing stream by default
            SurfaceTexture texture = mZegoEngine.getCustomVideoCaptureSurfaceTexture();
            texture.setDefaultBufferSize(mCaptureWidth, mCaptureHeight);
            //Create Surface through the obtained SurfaceTexture
            mSurface = new Surface(texture);
            //Through mSurface, complete sending screen recording data to ZEGO SDK
            mVirtualDisplay = mMediaProjection.createVirtualDisplay("ScreenCapture",
                    mCaptureWidth, mCaptureHeight, 1,
                    DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC, mSurface, null, mHandler);
        }
    }
}

//ZegoVideoCaptureCallback inherits from IZegoCustomVideoCaptureHandler
class VideoCaptureScreen extends ZegoVideoCaptureCallback {
    @Override
    //After receiving the onStart callback, you can create VirtualDisplay through MediaProjection and send screen data to ZEGO SDK
    public void onStart(ZegoPublishChannel channel) {
        if (mZegoEngine != null && !mIsCapturing && mMediaProjection != null) {
            mIsCapturing = true;
            //Get SurfaceTexture through ZEGO API getCustomVideoCaptureSurfaceTexture. This interface uses the main channel for pushing stream by default
            SurfaceTexture texture = mZegoEngine.getCustomVideoCaptureSurfaceTexture();
            texture.setDefaultBufferSize(mCaptureWidth, mCaptureHeight);
            //Create Surface through the obtained SurfaceTexture
            mSurface = new Surface(texture);
            //Through mSurface, complete sending screen recording data to ZEGO SDK
            mVirtualDisplay = mMediaProjection.createVirtualDisplay("ScreenCapture",
                    mCaptureWidth, mCaptureHeight, 1,
                    DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC, mSurface, null, mHandler);
        }
    }
}

So far, we have completed the operation of collecting screen data and sharing it to remote users through ZegoExpress SDK.

Watch remote screen sharing

After completing the above steps, other users can use the startPlayingStream interface to pull the screen sharing stream. For detailed steps, please refer to Quick Start.

// Similarly, users pulling and playing streams first need to initialize the SDK and login to the same room
...
...

// play stream and play, need to pass in the streamID used by the user who initiated screen sharing
mSDKEngine.startPlayingStream(streamID, new ZegoCanvas(playView));

// Similarly, users pulling and playing streams first need to initialize the SDK and login to the same room
...
...

// play stream and play, need to pass in the streamID used by the user who initiated screen sharing
mSDKEngine.startPlayingStream(streamID, new ZegoCanvas(playView));

Windows Implementation Process

Prerequisites

Before using screen sharing, please first download the ScreenCapture SDK plugin and integrate it in your project.

Usage Steps

Screen sharing supports the following functions. For detailed implementation process, please refer to the relevant documentation in the screen capture plugin:

macOS Implementation Process

Usage Steps

Screen sharing supports the following functions. For detailed implementation process, please refer to the relevant documentation in screen sharing: