Custom Audio Capture and Rendering

Overview

Custom Audio Capture

In the following scenarios, it is recommended to use the custom audio capture feature:

• Developers need to obtain captured input from existing audio streams, audio files, or customized capture systems and hand it over to the SDK for transmission.
• Developers have their own requirements for special audio effects processing on PCM input sources, and after audio effects processing, hand it over to the SDK for transmission.

Custom Audio Rendering

When developers have their own rendering requirements, such as performing special applications or processing on the captured raw PCM data before rendering, it is recommended to use the SDK's custom audio rendering feature.

Prerequisites

Before implementing custom audio capture and rendering, please ensure:

• You have created a project in the ZEGOCLOUD Console and obtained a valid AppID and AppSign. For details, please refer to Console - Project Information.
• You have integrated the ZEGO Express SDK into your project and implemented basic audio and video publishing and playing functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

The following figure shows the API call sequence diagram:

1 Initialize SDK

Please refer to "Create Engine" in Quick Start - Implementation.

2 Enable custom audio capture and rendering

Call the enableCustomAudioIO interface to enable custom audio IO functionality.

// Set audio source to custom capture and rendering
var config = ZegoCustomAudioConfig(ZegoAudioSourceType.Custom);
ZegoExpressEngine.instance.enableCustomAudioIO(true,config);

// Set audio source to custom capture and rendering
var config = ZegoCustomAudioConfig(ZegoAudioSourceType.Custom);
ZegoExpressEngine.instance.enableCustomAudioIO(true,config);

3 Login to room and publish/play streams

Please refer to "Login room", "Publish stream", and "Play stream" in Quick Start - Implementation.

4 Capture audio data

Open the audio capture device and pass the captured audio data to the engine through sendCustomAudioCaptureAACData or sendCustomAudioCapturePCMData.

5 Render audio data

Use fetchCustomAudioRenderPCMData to get the data to be rendered from the engine, and then play it through the rendering device after obtaining the audio data.

FAQ

1. When to call custom audio capture and rendering related interfaces?

   • enableCustomAudioIO: Should be called before the engine starts, that is, before starting preview, publishing, and playing streams.
   • sendCustomAudioCaptureAACData/sendCustomAudioCapturePCMData: Should be called after starting preview and publishing streams. If called before starting preview and publishing streams, the SDK will directly discard the received data.
   • fetchCustomAudioRenderPCMData: Should be called after starting to play stream. The data obtained before starting to play stream is all invalid mute data.

2. Frequency of calling custom audio capture and rendering related interfaces?

   The optimal way is to drive according to the clock of the physical audio device, call sendCustomAudioCaptureAACData and sendCustomAudioCapturePCMData when the physical capture device captures data; call fetchCustomAudioRenderPCMData when the physical rendering device needs data.

   If there is no specific physical device to drive in the actual scenario, it is recommended to call the above interfaces once every 10 ms to 20 ms.

3. When calling fetchCustomAudioRenderPCMData, if the internal data of the SDK is insufficient for "dataLength", how does the SDK handle it?

   When the "param" is filled normally, if the internal data of the SDK is insufficient for "dataLength", the remaining insufficient length will be filled with mute data.

4. Android device with external microphone, using custom audio capture and rendering, if the user puts on Bluetooth headphones midway, how to use Express SDK to capture audio?

   Since Express SDK will not automatically switch to internal capture, developers need to handle business logic: stop external capture. The mobile SDK will select devices based on the system's current route (audio route). If the system's route is Bluetooth, it will use Bluetooth for capture.