Screen Sharing

Overview

Screen Sharing refers to sharing screen content as a video with other viewers during video calls or interactive live streaming to enhance interactive experience and improve communication efficiency.

Screen Sharing is widely used in the following scenarios:

• In video conferencing scenarios, screen sharing allows the speaker to share local files, data, web pages, PPTs, and other visuals with other participants.
• In online classroom scenarios, screen sharing allows teachers to display courseware, notes, lecture content, and other visuals to students.

Download Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please check the files in the "lib\topics\OtherFunctions\screen_sharing" directory.

Prerequisites

Before implementing Screen Sharing functionality, please ensure:

• Platform and device requirements:
  • Android platform: Supports Android 5.0 or above, and Android devices or emulators that support audio and video (real devices are recommended).
  • iOS platform:
    • Supports iOS 12.0 or above, and iOS devices or emulators that support audio and video (real devices are recommended).
    • This feature has high performance requirements for devices, and it is recommended to use iPhone X and later models.
  • Windows platform: Windows 8, Windows 10 or above.
  • Web platform:
    • Screen sharing currently only supports desktop browsers such as Chrome, Edge, and Firefox on Windows and macOS, as well as Safari on macOS.
      • Plugin-based: Supports screen sharing in Chrome browser version 65 or above.
      • Plugin-free: Supports screen sharing in Chrome browser version 72 or above, Edge browser version 80 or above, and Firefox and Safari browsers version 13 or above.
    • If developers are using macOS, please first add the browser being used to "System Preferences > Security & Privacy > Screen Recording" before performing screen sharing.
• Have created a project in the ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
• Have integrated ZEGO Express SDK in the project and implemented basic audio/video Publish stream and Play stream functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation Flow.

Usage Steps

1 Enable Permissions

• (Required) On Android, a popup will prompt the user to allow the app to record the screen before recording. Please authorize.

• macOS requires enabling screen recording and accessibility permissions in "Security & Privacy" when calling screen sharing interfaces on macOS. If they don't take effect, you need to delete the previous permissions and add them again.
  • Screen recording permission

• Accessibility permission

• Install screen sharing plugin for Web platform (optional)

2 Windows and macOS need to get window (including screen) list information

The SDK can get all currently shareable window information through getScreenCaptureSources.

List<ZegoScreenCaptureSourceInfo> list = await ZegoExpressEngine.instance.getScreenCaptureSources(400, 400, 100, 100);

List<ZegoScreenCaptureSourceInfo> list = await ZegoExpressEngine.instance.getScreenCaptureSources(400, 400, 100, 100);

3 Create screen sharing source

Call the createScreenCaptureSource interface to create a screen sharing source object.

// For Android, iOS, and Web platforms, sourceId and sourceType do not need to be passed in, and passing them in will not cause creation to fail
ZegoScreenCaptureSource source = await ZegoExpressEngine.instance.createScreenCaptureSource();

// For Windows and macOS platforms, sourceId and sourceType are required parameters, and the data comes from the results obtained by the getScreenCaptureSources interface
// ZegoScreenCaptureSource source = await ZegoExpressEngine.instance.createScreenCaptureSource(sourceId: list[0].sourceID, sourceType: list[0].sourceType);

// For Android, iOS, and Web platforms, sourceId and sourceType do not need to be passed in, and passing them in will not cause creation to fail
ZegoScreenCaptureSource source = await ZegoExpressEngine.instance.createScreenCaptureSource();

// For Windows and macOS platforms, sourceId and sourceType are required parameters, and the data comes from the results obtained by the getScreenCaptureSources interface
// ZegoScreenCaptureSource source = await ZegoExpressEngine.instance.createScreenCaptureSource(sourceId: list[0].sourceID, sourceType: list[0].sourceType);

4 Set capture source to screen sharing source

To set the capture source to screen sharing source, you need to set both the video source and audio source.

Video Source

The default video source for SDK Publish stream is the camera source. If you need to Publish stream the screen sharing source, you need to switch to screen sharing through setVideoSource.

ZegoExpressEngine.instance.setVideoSource(ZegoVideoSourceType.ZegoVideoSourceScreenCapture, instanceID: source.getIndex(), channel: ZegoPublishChannel.Main);

ZegoExpressEngine.instance.setVideoSource(ZegoVideoSourceType.ZegoVideoSourceScreenCapture, instanceID: source.getIndex(), channel: ZegoPublishChannel.Main);

Audio Source

The default audio source for SDK Publish stream is the microphone source. If you need to Publish stream the screen sharing source, you need to switch to screen sharing through setAudioSource.

ZegoExpressEngine.instance.setAudioSource(ZegoAudioSourceType.ScreenCapture, channel: ZegoPublishChannel.Main);

ZegoExpressEngine.instance.setAudioSource(ZegoAudioSourceType.ScreenCapture, channel: ZegoPublishChannel.Main);

5 Start screen sharing

There are two screen sharing methods: in-app screen sharing and cross-app screen sharing.

Special configurations required before starting screen sharing on iOS platform

In-app screen sharing

If users only share video and audio within the app, they can call the startCapture interface to start screen sharing and set the inApp property to true. You can also call the broadcastFinished interface callback to receive screen sharing end notifications. If screen capture fails, you can receive the reason for the failure.

var config = ZegoScreenCaptureConfig(true, true, 100, 100);
source.startCapture(config: config, inApp: true);

var config = ZegoScreenCaptureConfig(true, true, 100, 100);
source.startCapture(config: config, inApp: true);

Cross-app screen sharing

Cross-app screen sharing is recorded by the iOS system through an Extension process, so you need to additionally create and start an Extension process. For details, please refer to the following implementation steps:

1

Create Broadcast Upload Extension (iOS platform)

Note

The memory usage limit for Broadcast Upload Extension is 50 MB. Please do not perform additional memory allocation in the screen sharing Extension.

1. Use Xcode to open the iOS native project file with the suffix .xcworkspace in the iOS directory under your Flutter project, then click "File > New > Target..." in the menu bar in sequence.

2. Select "Broadcast Upload Extension" on the iOS page in the pop-up window, then click "Next".

3. Fill in the name of the "Broadcast Upload Extension" in the "Product Name" field in the pop-up dialog, such as "ScreenShare". After selecting "Team", "Language" and other information, click "Finish".

Note

There is no need to check "Include UI Extension".

After creation, you will see the folder of this Extension in the project. The structure is similar to the following. This folder is used to store the implementation code of the screen sharing function:

2

Configure Extension

1. Ensure that in the "Info.plist" file of the Extension, "RPBroadcastProcessMode" is set to "RPBroadcastProcessModeSampleBuffer".

2. Import ZEGO Express SDK in the Extension. Click the project in Xcode to enter the project configuration page. Select the Extension created above, find the Frameworks and Libraries item under General, click the + sign, and select ZegoExpressEngine.xcframework to add.

   

3

Start screen sharing

If users need to share the entire system's video and audio, they can call the startCapture interface to start screen sharing.

var config = ZegoScreenCaptureConfig(true, true, 100, 100);
source.startCapture(config: config, inApp: false);

var config = ZegoScreenCaptureConfig(true, true, 100, 100);
source.startCapture(config: config, inApp: false);

4

Get screen data

Note

The implementation of the following system callbacks can be viewed in the "/ios/ScreenShare/SampleHandler.m" file in the SDK source code:

• broadcastStartedWithSetupInfo
• processSampleBuffer
• broadcastFinished

1. The system notifies the Extension through the broadcastStartedWithSetupInfo callback that it has started recording the screen. You need to call the following interface in the ZegoReplayKitExt class in this callback to create a data transmission channel:

   [ZegoReplayKitExt.sharedInstance setupWithDelegate:self];

   [ZegoReplayKitExt.sharedInstance setupWithDelegate:self];

2. In the processSampleBuffer system callback, send to ZEGO Express SDK through the sendSampleBuffer interface in the ZegoReplayKitExt class.

   [ZegoReplayKitExt.sharedInstance sendSampleBuffer:sampleBuffer withType:sampleBufferType];

   [ZegoReplayKitExt.sharedInstance sendSampleBuffer:sampleBuffer withType:sampleBufferType];

3. The system notifies the Extension through the broadcastFinished callback that screen recording has ended. You can call the following interface in the ZegoReplayKitExt class in this callback to stop screen sharing and disconnect the data transmission channel:

   [ZegoReplayKitExt.sharedInstance finished];

   [ZegoReplayKitExt.sharedInstance finished];

Call the startCapture interface to share the entire system's screen.

source.startCapture();

source.startCapture();

source.updateCaptureSource(list[1].sourceID, list[1].sourceType)

source.updateCaptureSource(list[1].sourceID, list[1].sourceType)

source.updateCaptureRegion(Rect.fromLTWH(10, 10, 400, 400));

source.updateCaptureRegion(Rect.fromLTWH(10, 10, 400, 400));

source.setExcludeWindowList([list[1].sourceID]);

source.setExcludeWindowList([list[1].sourceID]);

source.enableWindowActivate(true);

source.enableWindowActivate(true);

source.enableCursorVisible(true);

source.enableCursorVisible(true);

ZegoExpressEngine.onExceptionOccurred = (ZegoScreenCaptureSource source,
      ZegoScreenCaptureSourceExceptionType exceptionType) {
};

ZegoExpressEngine.onExceptionOccurred = (ZegoScreenCaptureSource source,
      ZegoScreenCaptureSourceExceptionType exceptionType) {
};

var config = ZegoScreenCaptureConfig(true, true, 100, 100);
config.orientation = ZegoScreenCaptureOrientation.Landscape; // Fixed landscape
source.startCapture(config: config);
// Or set when updating screen sharing configuration
source.updateScreenCaptureConfig(config);

var config = ZegoScreenCaptureConfig(true, true, 100, 100);
config.orientation = ZegoScreenCaptureOrientation.Landscape; // Fixed landscape
source.startCapture(config: config);
// Or set when updating screen sharing configuration
source.updateScreenCaptureConfig(config);

ZegoExpressEngine.onMobileScreenCaptureStart = () {
};
ZegoExpressEngine.onMobileScreenCaptureExceptionOccurred = (ZegoScreenCaptureSourceExceptionType exceptionType) {
};

ZegoExpressEngine.onMobileScreenCaptureStart = () {
};
ZegoExpressEngine.onMobileScreenCaptureExceptionOccurred = (ZegoScreenCaptureSourceExceptionType exceptionType) {
};

6 Login to room and start Publish stream

Call the loginRoom interface, pass in the room ID parameter "roomID" and user parameter "user", to login to the room.

Call the startPublishingStream interface, pass in the stream ID parameter "streamID", to send the local audio/video stream to remote users.

// Create user
var user = new ZegoUser.id("user1");

// Start login to room
ZegoExpressEngine.instance.loginRoom("room1", user);
// Start Publish stream
ZegoExpressEngine.instance.startPublishingStream("stream1");

// Create user
var user = new ZegoUser.id("user1");

// Start login to room
ZegoExpressEngine.instance.loginRoom("room1", user);
// Start Publish stream
ZegoExpressEngine.instance.startPublishingStream("stream1");

At this point, we have completed the operation of capturing screen data and sharing it to remote users through ZEGO Express SDK.

7 Watch remote screen sharing

After completing the above steps, other users can use the startPlayingStream interface to Play stream the screen sharing stream.

// Play stream, need to pass in the streamID used when the user who initiated screen sharing Publish stream
ZegoExpressEngine.instance.startPlayingStream(streamID, canvas: ZegoCanvas.view(playView));

// Play stream, need to pass in the streamID used when the user who initiated screen sharing Publish stream
ZegoExpressEngine.instance.startPlayingStream(streamID, canvas: ZegoCanvas.view(playView));

8 Stop screen sharing

Call the stopCapture interface to stop sharing.

source.stopCapture();

source.stopCapture();

More Features

Border

When screen sharing, you can call the enableHightLight interface to enable border for screen regions or windows, and set the border color and width.

// Border size, default value is 4, maximum value is 100
// Border color, format is 0xRRGGBB, default is green, i.e., 0x00FF00
var config = ZegoLayerBorderConfig(4, 0x00FF00);
source.enableHightLight(true, config)

// Border size, default value is 4, maximum value is 100
// Border color, format is 0xRRGGBB, default is green, i.e., 0x00FF00
var config = ZegoLayerBorderConfig(4, 0x00FF00);
source.enableHightLight(true, config)

FAQ

When Android device is fixed in portrait mode (auto-rotation disabled), the popup orientation is abnormal and the interface switches between landscape and portrait when starting screen capture. How to solve?

Due to the conflict between the device fixed in portrait mode and the App landscape interface, rapid switching between landscape and portrait occurs during screen capture, causing system data abnormalities and black screen. Solutions:

• Solution 1: Override SDK configuration through tools:replace

  The default orientation of the SDK screen sharing auxiliary Activity is "fullUser". You need to modify it in the project's AndroidManifest.xml:

  1. Add the "tools" tool namespace to the root node (manifest tag);
  2. Find the above Activity configuration, change its screen orientation (screenOrientation) to "unspecified", and add tools:replace to forcibly override the SDK default configuration. After recompiling, the popup will be consistent with the App landscape, avoiding switching and black screen.

• Solution 2: Enable device auto-rotation

  Go to device "Settings > Display > Auto-rotate screen". After enabling, the popup orientation will follow the App landscape, and there will be no switching and black screen issue.