Custom Audio Capture and Rendering

Introduction

Custom Audio Capture

In the following situations, it is recommended to use the custom audio capture feature:

1. Customers need to get captured input from existing audio streams, audio files, or customized capture systems and hand it over to the SDK for transmission.
2. Customers have their own needs to do special audio effect processing on PCM input sources, and input after audio effect processing, handing it over to the SDK for transmission.

Custom Audio Rendering

When customers have their own rendering needs, for example, doing special applications or processing on the obtained raw PCM data before rendering, it is recommended to use the SDK's custom audio rendering feature.

Download Sample Source Code

Please refer to Download Sample Source Code to obtain the source code.

For related source code, please check the files in the "/ZegoExpressExample/Examples/AdvancedAudioProcessing/CustomAudioCaptureAndRendering" directory.

Usage Steps

The following figure is the API interface call sequence diagram:

1 Initialize the SDK

Please refer to "Create the engine" in Quick Start - Implementation Process.

2 Enable custom audio capture and rendering

You can call ZegoCustomAudioConfig to set sourceType = ZEGO_AUDIO_SOURCE_TYPE_CUSTOM, and then call the enableCustomAudioIO interface to enable the custom audio IO feature.

Call example:

// Set audio source to custom capture and rendering
ZegoCustomAudioConfig audioConfig;
audioConfig.sourceType = ZEGO_AUDIO_SOURCE_TYPE_CUSTOM;
engine->enableCustomAudioIO(true, &audioConfig);

// Set audio source to custom capture and rendering
ZegoCustomAudioConfig audioConfig;
audioConfig.sourceType = ZEGO_AUDIO_SOURCE_TYPE_CUSTOM;
engine->enableCustomAudioIO(true, &audioConfig);

3 Login to room and then publish/play streams

Please refer to "Login room", "Publish stream", and "Play stream" in Quick Start - Implementation Process.

4 Capture audio data

Pass the captured audio data to the engine through sendCustomAudioCaptureAACData or sendCustomAudioCapturePCMData.

5 Render audio data

Use fetchCustomAudioRenderPCMData to get audio data from the engine, and then play it through rendering devices after getting the audio data.

FAQ

1. Timing for calling custom audio capture and rendering related interfaces?

   • enableCustomAudioIO: Should be called before starting the engine, that is, before starting preview, publishing, and playing streams.
   • sendCustomAudioCaptureAACData/sendCustomAudioCapturePCMData: Should be called after starting preview and publishing streams. If called before starting preview and publishing streams, the SDK will directly discard the received data.
   • fetchCustomAudioRenderPCMData: Should be called after starting to play stream. Data obtained before starting to play stream is invalid silent data.

2. Frequency for calling custom audio capture and rendering related interfaces?

   The optimal way is to drive according to the clock of physical audio devices, call sendCustomAudioCaptureAACData and sendCustomAudioCapturePCMData when the physical capture device captures data; call fetchCustomAudioRenderPCMData when the physical rendering device needs data. If there is no specific physical device to drive in the developer's actual scenario, it is recommended to call the above interfaces every 10 ~ 20 ms.

3. When calling fetchCustomAudioRenderPCMData, if the data inside the SDK is insufficient for "dataLength", how does the SDK handle it?

   When "param" is filled normally, if the data inside the SDK is insufficient for "dataLength", the remaining insufficient length will be filled with silent data.