Custom Audio Processing
Introduction
Custom audio processing is generally used to remove interference from speech. Since the SDK has already processed the captured raw audio data with acoustic echo cancellation (AEC), noise suppression, etc., developers usually do not need to process it again. If developers want to implement special functions through custom processing after capturing audio data or before rendering remote audio data (such as voice changing, beautification, etc.), they can refer to this document.
The data for custom audio processing is audio data after 3A (AEC Acoustic Echo Cancellation, AGC Automatic Gain Control, ANS Noise Suppression) processing on the original audio:
- If developers need to process the raw data, please first call the enableAEC, enableAGC, and enableANS interfaces to disable audio 3A processing. If audio effects processing such as voice changing, reverb, and stereo sound is enabled (disabled by default), they also need to be disabled first before raw audio data can be obtained.
- If developers need to obtain both raw data and audio data after 3A processing for processing, please refer to Custom Audio Capture and Rendering.
Usage Steps
1 Create the SDK engine
Call the createEngine interface to create an SDK engine instance. For details, please refer to "Create the engine" in Quick Start - Implementation Process.
ZegoEngineProfile profile;
profile.appID = ZegoUtilHelper::AppID();
profile.appSign = ZegoUtilHelper::AppSign();
profile.scenario = ZegoScenario::ZEGO_SCENARIO_DEFAULT;
ZegoExpressSDK::createEngine(profile, nullptr);2 Set the audio custom processing handler and implement callback methods
Call the setCustomAudioProcessHandler interface to set the audio custom processing handler, and implement the callback methods: custom audio processing local captured PCM audio frame callback onProcessCapturedAudioData and custom audio processing remote stream playing PCM audio frame callback onProcessRemoteAudioData. By directly processing the obtained data in the callback method, you can implement processing of published and played audio data.
class MyAudioProcessHandler: public IZegoCustomAudioProcessHandler{
public:
void onProcessCapturedAudioData(unsigned char* data, unsigned int dataLength, ZegoAudioFrameParam* param, double timestamp) override{
}
void onProcessRemoteAudioData(unsigned char* data, unsigned int dataLength, ZegoAudioFrameParam* param, const std::string& streamID, double timestamp) override {
}
};
engine->setCustomAudioProcessHandler(std::make_shared<MyAudioProcessHandler>());3 Custom audio processing
-
Before starting to publish stream or starting local preview, call the enableCustomAudioCaptureProcessing interface to enable custom audio processing for local capture. After enabling, developers can receive locally captured audio frames through the onProcessCapturedAudioData callback and can modify the audio data.
-
Before starting to play stream, call the enableCustomAudioRemoteProcessing interface to enable custom audio processing for remote stream playing. After enabling, developers can receive remote stream audio frames through onProcessRemoteAudioData and can modify the audio data.
ZegoCustomAudioProcessConfig config;
config.channel = ZEGO_AUDIO_CHANNEL_MONO;
config.samples = 2048;
config.sampleRate = ZEGO_AUDIO_SAMPLE_RATE_44K;
engine->enableCustomAudioRemoteProcessing(true, &config);
engine->enableCustomAudioCaptureProcessing(true, &config);