Audio Mixing

2024-01-30

Feature Overview

Audio mixing refers to SDK integrating third-party audio data with microphone-Captured audio data into one audio stream, to achieve playing sound effects (such as applause, whistles), custom sounds, music files, etc. during calls or live streaming, and allowing other members in the Room to hear the sound.

Example Source Code Download

Please refer to Download Example Source Code to get the source code.

For related source code, please check files in the "src/Examples/AdvancedAudioProcessing/AudioMixing" directory.

Prerequisites

Before using audio mixing functionality, please ensure:

A project has been created in ZEGOCLOUD Console, and valid AppID and Server address have been obtained. For details, please refer to Console - Project Information.
ZEGO Express SDK has been integrated into the project, and basic audio/video streaming functionality has been implemented. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

(Optional) Pitch Shifting

Before starting audio mixing, developers can choose whether to change the pitch of the media tag's audio as needed

Before audio mixing, developers can change the pitch of audio data through the setAudioChangerParam interface according to business needs. For example, in KTV solo scenarios, raise or lower the pitch of accompaniment while keeping the human voice unchanged.

Warning

At the same time, data from one <video> or <audio> tag can only undergo one type of audio effect processing. Multiple types of processing cannot be performed simultaneously, i.e., voice changing, reverb, stereo, and pitch shifting can only choose one to use.
For audio data from the same <audio> tag, if the developer calls setAudioChangerParam and enableLiveAudioEffect (both interfaces contain a mode parameter for setting the audio effect activation mode), the audio effect activation mode will be based on the setting of the last called interface.

const result = await zg.setAudioChangerParam(
    document.getElementById("audio"),
    3,
    3
);

const result = await zg.setAudioChangerParam(
    document.getElementById("audio"),
    3,
    3
);

Start Audio Mixing

After publishing the stream, call the startMixingAudio method to start audio mixing, and mix audio into the stream being published through streamID and mediaList (media tag list: video or audio).

Warning

Control pause and resume operations of already mixed audio through video or audio media tags.
Since Chrome 86, when mixing third-party audio, if local audio is set to mute, the playing end will not be able to hear the corresponding sound.
Affected by Safari browser policy, setting the audio tag to mute will result in automatic pause after playback and inability to mix multiple audio tracks.
If the capability detection interface checkSystemRequirements detects that customCapture is false, i.e., the browser does not support obtaining MediaStream from media elements, this interface cannot be used.

const result = zg.startMixingAudio(publishStreamId, [
    $('#extenerVideo1')[0] ,
    $('#extenerVideo2')[0] ,
]);
console.warn('Audio mixing', result);

const result = zg.startMixingAudio(publishStreamId, [
    $('#extenerVideo1')[0] ,
    $('#extenerVideo2')[0] ,
]);
console.warn('Audio mixing', result);

Stop Audio Mixing

Warning

If the capability detection interface checkSystemRequirements detects that customCapture is false, i.e., the browser does not support obtaining MediaStream from media elements, this interface cannot be used.

Call the stopMixingAudio method to stop audio mixing.

zg.stopMixingAudio(publishStreamId);

zg.stopMixingAudio(publishStreamId);