Screen Sharing

Overview

Screen sharing refers to sharing screen content with other audiences as video during video calls or interactive live streaming to enhance interactive experience and improve communication efficiency.

Screen sharing is widely used in the following scenarios:

• In video conference scenarios, screen sharing can share the speaker's local files, data, web pages, PPT, and other pictures with other participants;
• In online classroom scenarios, screen sharing can display the teacher's courseware, notes, lecture content, and other pictures to students.

Prerequisites

Before implementing screen sharing functionality, please ensure:

• Supports Android 5.0 or above, and Android devices or emulators that support audio and video (real devices are recommended).
• Supports iOS 12.0 or above, and iOS devices or emulators that support audio and video (real devices are recommended).
• This function has high requirements on device performance. It is recommended to use it on iPhone X and later models.
• You have created a project in the ZEGO Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information.
• You have integrated ZEGO Express SDK into the project and implemented basic audio and video streaming functionality. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

Enable Permissions

(Required) Before recording the screen on Android, a pop-up will prompt the user to allow the application to record the screen. Please authorize.

If the target Android SDK version is 34.0.0 or later, you need to go to the "app/src/main" directory of your project, open the "AndroidManifest.xml" file, and set the FOREGROUND_SERVICE and FOREGROUND_SERVICE_MEDIA_PROJECTION permission declarations.

<uses-permission android:name="android.permission.FOREGROUND_SERVICE"/>
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION"/>

<uses-permission android:name="android.permission.FOREGROUND_SERVICE"/>
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION"/>

Set Capture Source to Screen Sharing Source

Video Source

The default video source for SDK publishing stream is the camera source. If you need to publish screen sharing source, you need to switch to screen sharing through setVideoSource.

ZegoExpressEngine.instance().setVideoSource(ZegoVideoSourceType.ScreenCapture, ZegoPublishChannel.Aux);

ZegoExpressEngine.instance().setVideoSource(ZegoVideoSourceType.ScreenCapture, ZegoPublishChannel.Aux);

Audio Source (iOS Only)

The default audio source for SDK publishing stream is the microphone source. If you need to publish screen sharing source, you need to switch to screen sharing through setAudioSource.

ZegoExpressEngine.instance().setAudioSource(ZegoAudioSourceType.ScreenCapture, ZegoPublishChannel.Aux);

ZegoExpressEngine.instance().setAudioSource(ZegoAudioSourceType.ScreenCapture, ZegoPublishChannel.Aux);

Start Screen Sharing

There are two screen sharing methods: in-app screen sharing and cross-app screen sharing.

In-App Screen Sharing (iOS Only)

If users only share pictures and sound within the app, they can call the startScreenCaptureInApp interface to start screen sharing. You can also call the broadcastFinished interface callback to receive screen sharing end notifications. If screen capture fails, the reason for failure can be received.

ZegoExpressEngine.instance().startScreenCaptureInApp();

ZegoExpressEngine.instance().startScreenCaptureInApp();

Cross-App Screen Sharing

Cross-app screen sharing is recorded by the iOS system through the Extension extension process, so you need to additionally create and start the extension process. For details, please refer to the following implementation steps:

Create Broadcast Upload Extension (iOS Only)

1. Use Xcode to open the project engineering file, and click "File > New > Target..." in the menu bar in sequence.

2. In the pop-up window, select "Broadcast Upload Extension" on the iOS page, and then click "Next".

3. In the "Product Name" field of the pop-up dialog box, fill in the name of the "Broadcast Upload Extension", such as "ScreenShare". After selecting "Team", "Language" and other information, click "Finish".

After creation, you will see the folder of this Extension in the project. The structure is similar to the following. This folder is used to store the implementation code of the screen sharing function:

Set Up Extension

1. Ensure that in the "Info.plist" file of the Extension, set "RPBroadcastProcessMode" to "RPBroadcastProcessModeSampleBuffer".
2. Import ZEGO Express SDK in the Extension. In xcode, click the project to enter the project configuration page. Select the Extension created above, find the Frameworks and Libraries item under General, click the + sign, and select ZegoExpressEngine.xcframework to add.

Start Screen Sharing

If users need to share the entire system, they can call the startScreenCapture interface to start screen sharing.

ZegoExpressEngine.instance().startScreenCapture();

ZegoExpressEngine.instance().startScreenCapture();

Get Screen Data

1. The system notifies the Extension through the broadcastStartedWithSetupInfo callback that screen recording has started. You need to call the following interface in the ZegoReplayKitExt class in this callback to create a data transmission channel:

   [ZegoReplayKitExt.sharedInstance setupWithDelegate:self];

   [ZegoReplayKitExt.sharedInstance setupWithDelegate:self];

2. In the processSampleBuffer system callback, send to ZEGO Express SDK through the sendSampleBuffer interface in the ZegoReplayKitExt class.

   [ZegoReplayKitExt.sharedInstance sendSampleBuffer:sampleBuffer withType:sampleBufferType];

   [ZegoReplayKitExt.sharedInstance sendSampleBuffer:sampleBuffer withType:sampleBufferType];

3. The system notifies the Extension through the broadcastFinished callback that screen recording has ended. You can call the following interface in the ZegoReplayKitExt class in this callback to stop screen sharing and disconnect the data transmission channel:

   [ZegoReplayKitExt.sharedInstance finished];

   [ZegoReplayKitExt.sharedInstance finished];

Login Room and Publish Stream

After completing the above screen sharing source capture process, startPublishingStream pushes the captured data source to the cloud server. (The channel for pushing the data source must be consistent with the channel set for the capture source)

ZegoExpressEngine.instance().startPublishingStream(streamID, ZegoPublishChannel.Aux);

ZegoExpressEngine.instance().startPublishingStream(streamID, ZegoPublishChannel.Aux);

Watch Remote Screen Sharing

After completing the above steps, other users can use the startPlayingStream interface to play the screen sharing stream.

// Play stream, need to pass in the streamID used when the user who initiated screen sharing published the stream
ZegoExpressEngine.instance().startPlayingStream(streamID, {"reactTag": findNodeHandle(this.refs.zego_play_view), "viewMode": 0, "backgroundColor": 0});

// Play stream, need to pass in the streamID used when the user who initiated screen sharing published the stream
ZegoExpressEngine.instance().startPlayingStream(streamID, {"reactTag": findNodeHandle(this.refs.zego_play_view), "viewMode": 0, "backgroundColor": 0});

Stop Screen Sharing

Call the stopScreenCapture interface to stop screen sharing.

ZegoExpressEngine.instance().stopScreenCapture();

ZegoExpressEngine.instance().stopScreenCapture();

FAQ

1. Does iOS support sharing a specified area?

   The iOS system only supports sharing the entire screen and does not support sharing a specified area.

2. Why does capture stop when iOS uses screen sharing and enters the background?

   • Enable the background mode for audio recording in the application.
   
   • If you use the main channel for screen sharing, the SDK will not start audio internal capture and cannot keep alive in the background. When the application goes to the background, screen sharing may stop. It is recommended to add background keep-alive processing logic to the application.

3. How to handle audio playback abnormalities when iOS uses screen sharing?

   If you use the screen sharing function to capture and publish audio while using the stream playing function locally, it will cause the iOS system to repeatedly capture the played audio, resulting in audio playback abnormalities. It is recommended to use muteAllPlayStreamAudio to prohibit playing all audio streams.

4. When the Android device is fixed in portrait mode (auto-rotation is prohibited), how to solve the problem of abnormal popup orientation and interface switching between portrait and landscape when screen capture is started?

   Due to the conflict between the device fixed in portrait mode and the app landscape interface, the screen capture causes rapid switching between portrait and landscape, triggering system data abnormality and black screen. Solutions:

   • Solution 1: Override SDK configuration through tools:replace

     The default orientation of the SDK screen sharing auxiliary Activity is "fullUser". You need to modify it in the project AndroidManifest.xml:

     1. Add the "tools" tool namespace at the root node (manifest tag);
     2. Find the above Activity configuration, change its screen orientation (screenOrientation) to "unspecified", and add tools:replace to force override the SDK default configuration. After recompiling, the popup will be consistent with the app landscape, avoiding switching black screen.

   • Solution 2: Enable device auto-rotation

     Go to device "Settings > Display > Auto-rotate screen". After enabling, the popup orientation will follow the app landscape, and there will be no switching black screen problem.