Custom Audio Capture and Rendering

2024-02-27

Feature Overview

Custom Audio Capture

It is recommended to use the custom audio capture function in the following situations:

Customers need to obtain captured input from existing audio streams, audio files, or customized collection systems, and hand it over to the SDK for transmission.
Customers have their own requirements for special sound effect processing on PCM input sources, and input after sound effect processing for SDK transmission.

Custom Audio Rendering

When customers have their own rendering requirements, such as special applications or processing on the captured raw PCM data before rendering, it is recommended to use the SDK's custom audio rendering function.

Warning

Audio capture and rendering are divided into 3 situations:

Internal capture, internal rendering
Custom capture, custom rendering
Custom capture, internal rendering

When developing on the Linux platform, you must have audio capture and rendering devices (sound cards) to use the SDK's internal capture and internal rendering functions; otherwise, you can only use custom capture and custom rendering. Developers should choose appropriate audio capture and rendering methods according to their own business scenarios.

Usage Steps

The following figure shows the API interface call sequence diagram:

1 Initialize SDK

Please refer to "Create Engine" in Quick Start - Implementation Flow.

2 Enable Custom Audio Capture and Rendering

Warning

enableCustomAudioIO needs to be called before startPublishingStream, startPlayingStream, startPreview, createMediaPlayer, createAudioEffectPlayer, and createRealTimeSequentialDataManager to take effect.

You can call ZegoCustomAudioConfig to set sourceType = ZEGO_AUDIO_SOURCE_TYPE_CUSTOM, then call the enableCustomAudioIO interface to enable custom audio IO function.

Call example:

// Set audio source to custom capture and rendering
ZegoCustomAudioConfig audioConfig;
audioConfig.sourceType = ZEGO_AUDIO_SOURCE_TYPE_CUSTOM;
engine->enableCustomAudioIO(true, &audioConfig);

// Set audio source to custom capture and rendering
ZegoCustomAudioConfig audioConfig;
audioConfig.sourceType = ZEGO_AUDIO_SOURCE_TYPE_CUSTOM;
engine->enableCustomAudioIO(true, &audioConfig);

Please refer to "Login Room", "Publish Stream", and "Play Stream" in Quick Start - Implementation Flow.

4 Capture Audio Data

Pass the captured audio data to the engine through sendCustomAudioCaptureAACData or sendCustomAudioCapturePCMData.

5 Render Audio Data

Use fetchCustomAudioRenderPCMData to obtain audio data from the engine, and then play it through the rendering device after obtaining the audio data.

FAQ

When to call custom audio capture and rendering related interfaces?
- enableCustomAudioIO: Should be called before the engine starts, that is, before starting preview, publishing and playing streams.
- sendCustomAudioCaptureAACData/sendCustomAudioCapturePCMData: Should be called after starting preview and publishing stream. If called before starting preview and publishing stream, the SDK will directly discard the received data.
- fetchCustomAudioRenderPCMData: Should be called after starting playing stream. Data obtained before starting playing stream is invalid mute data.
Frequency of calling custom audio capture and rendering related interfaces?

The optimal way is to drive according to the clock of the physical audio device, calling sendCustomAudioCaptureAACData and sendCustomAudioCapturePCMData when the physical capture device captures data; calling fetchCustomAudioRenderPCMData when the physical rendering device needs data. If there is no specific physical device to drive in the actual scenario, it is recommended to call the above interfaces once every 10 ~ 20 ms.
When calling fetchCustomAudioRenderPCMData, if the data inside the SDK is less than dataLength, how does the SDK handle it?

Under the condition that param is filled normally, when the data inside the SDK is less than dataLength, the remaining insufficient length will be filled with mute data.

Custom Audio Capture and Rendering

Feature Overview