Sound Level and Audio Spectrum

2024-01-31

Feature Overview

Concept	Description	Application Scenario	Scenario Diagram
Sound Level	Refers to the volume level of a stream, hereinafter referred to as "sound level".	During publishing and playing streams, determine which user on the mic is speaking and display it on the UI.
Audio Spectrum	Refers to the energy value of digital audio signals at each frequency point.	In KTV scenarios, on the premise of publishing or playing streams, let the anchor or audience see the animation of pitch and volume changes.

Usage Steps for Non-Stream Mixing Scenarios

1 Enable Sound Level and Spectrum Monitoring

The SDK disables sound level and spectrum monitoring by default. Users need to actively call related interfaces to enable monitoring. You can start listening to corresponding callbacks for sound level or audio spectrum respectively.

Interface prototype:

// Start sound level monitoring
virtual void startSoundLevelMonitor() = 0;

// Start audio spectrum monitoring
virtual void startAudioSpectrumMonitor() = 0;

// Start sound level monitoring
virtual void startSoundLevelMonitor() = 0;

// Start audio spectrum monitoring
virtual void startAudioSpectrumMonitor() = 0;

Call example:

Call the startSoundLevelMonitor interface to start listening to sound level:

// Start sound level monitoring
engine->startSoundLevelMonotior();

// Start sound level monitoring
engine->startSoundLevelMonotior();

Call the startAudioSpectrumMonitor interface to start listening to audio spectrum:

// Start audio spectrum monitoring
engine->startAudioSpectrumMonitor();

// Start audio spectrum monitoring
engine->startAudioSpectrumMonitor();

2 Listen to Sound Level and Spectrum Callbacks

After enabling sound level and spectrum monitoring, the SDK will periodically notify users of current sound level and spectrum data through related callbacks (onCapturedSoundLevelUpdate , onRemoteSoundLevelUpdate , onCapturedAudioSpectrumUpdate , onRemoteAudioSpectrumUpdate ). Users only need to override related callback functions to implement UI display.

Interface prototype:

/**
* Local captured audio sound level callback
* @param soundLevel Locally captured sound level value, ranging from 0.0~100.0
*/
virtual void onCapturedSoundLevelUpdate(double soundLevel) {

}

/**
* Remote audio sound level callback
* @param soundLevels Remote sound level key-value pairs, key is stream ID, value is the sound level value of the corresponding stream
*/
virtual void onRemoteSoundLevelUpdate(const std::map<std::string, double>& soundLevels) {

}

/**
* Local captured audio spectrum callback
* @param audioSpectrum Locally captured audio spectrum value array, spectrum value range is [0-2^30]
*
*/
virtual void onCapturedAudioSpectrumUpdate(const ZegoAudioSpectrum& audioSpectrum) {

}

/**
* Remote playing stream audio spectrum callback
* @param audioSpectrums Remote audio spectrum key-value pairs, key is stream ID, value is the audio spectrum value array of the corresponding stream, spectrum value range is [0-2^30]
*
*/
virtual void onRemoteAudioSpectrumUpdate(const std::map<std::string, ZegoAudioSpectrum>& audioSpectrums) {

}

/**
* Local captured audio sound level callback
* @param soundLevel Locally captured sound level value, ranging from 0.0~100.0
*/
virtual void onCapturedSoundLevelUpdate(double soundLevel) {

}

/**
* Remote audio sound level callback
* @param soundLevels Remote sound level key-value pairs, key is stream ID, value is the sound level value of the corresponding stream
*/
virtual void onRemoteSoundLevelUpdate(const std::map<std::string, double>& soundLevels) {

}

/**
* Local captured audio spectrum callback
* @param audioSpectrum Locally captured audio spectrum value array, spectrum value range is [0-2^30]
*
*/
virtual void onCapturedAudioSpectrumUpdate(const ZegoAudioSpectrum& audioSpectrum) {

}

/**
* Remote playing stream audio spectrum callback
* @param audioSpectrums Remote audio spectrum key-value pairs, key is stream ID, value is the audio spectrum value array of the corresponding stream, spectrum value range is [0-2^30]
*
*/
virtual void onRemoteAudioSpectrumUpdate(const std::map<std::string, ZegoAudioSpectrum>& audioSpectrums) {

}

Call example:

class MyEventHandler: public IZegoEventHandler
{
    virtual void onCapturedSoundLevelUpdate(double soundLevel) {
        printf("onCapturedSoundLevelUpdate");
        ... // operate ui
    }

    virtual void onRemoteSoundLevelUpdate(const std::map<std::string, double>& soundLevels) {
        printf("onRemoteSoundLevelUpdate");
        ... // operate ui
    }

    virtual void onCapturedAudioSpectrumUpdate(const ZegoAudioSpectrum& audioSpectrum) {
        printf("onCapturedAudioSpectrumUpdate");
        ... // operate ui
    }

    virtual void onRemoteAudioSpectrumUpdate(const std::map<std::string, ZegoAudioSpectrum>& audioSpectrums) {
        printf("onRemoteAudioSpectrumUpdate");
        ... // operate ui
    }
};

class MyEventHandler: public IZegoEventHandler
{
    virtual void onCapturedSoundLevelUpdate(double soundLevel) {
        printf("onCapturedSoundLevelUpdate");
        ... // operate ui
    }

    virtual void onRemoteSoundLevelUpdate(const std::map<std::string, double>& soundLevels) {
        printf("onRemoteSoundLevelUpdate");
        ... // operate ui
    }

    virtual void onCapturedAudioSpectrumUpdate(const ZegoAudioSpectrum& audioSpectrum) {
        printf("onCapturedAudioSpectrumUpdate");
        ... // operate ui
    }

    virtual void onRemoteAudioSpectrumUpdate(const std::map<std::string, ZegoAudioSpectrum>& audioSpectrums) {
        printf("onRemoteAudioSpectrumUpdate");
        ... // operate ui
    }
};

3 Stop Sound Level and Spectrum Monitoring

You can stop listening to corresponding callbacks for sound level or audio spectrum respectively.

Interface prototype:

// Stop sound level monitoring
virtual void stopSoundLevelMonitor() = 0;
// Stop audio spectrum monitoring
virtual void stopAudioSpectrumMonitor() = 0;

// Stop sound level monitoring
virtual void stopSoundLevelMonitor() = 0;
// Stop audio spectrum monitoring
virtual void stopAudioSpectrumMonitor() = 0;

Call example:

Call the stopSoundLevelMonitor interface to stop listening to sound level:

// Stop sound level monitoring
engine->stopSoundLevelMonotior();

// Stop sound level monitoring
engine->stopSoundLevelMonotior();

Call the stopAudioSpectrumMonitor interface to stop listening to audio spectrum:

// Stop audio spectrum monitoring
engine->stopAudioSpectrumMonitor();

// Stop audio spectrum monitoring
engine->stopAudioSpectrumMonitor();

Usage Steps for Stream Mixing Scenarios

1 Listen to Stream Mixing Sound Level Callback Interface

Interface prototype

Sound level update callback interface for each single stream in manual stream mixing onMixerSoundLevelUpdate:

/**
 * Sound level update callback for each single stream in stream mixing
 *
 * Callback notification period is 100 ms.
 * @param soundLevel Sound level key-value pairs for each single stream in stream mixing, key is the soundLevelID of each single stream, value is the sound level value of the corresponding single stream. Value range: value ranges from 0.0 to 100.0.
 */
public void onMixerSoundLevelUpdate(const std::unordered_map<unsigned int, float> &soundLevels){

}

/**
 * Sound level update callback for each single stream in stream mixing
 *
 * Callback notification period is 100 ms.
 * @param soundLevel Sound level key-value pairs for each single stream in stream mixing, key is the soundLevelID of each single stream, value is the sound level value of the corresponding single stream. Value range: value ranges from 0.0 to 100.0.
 */
public void onMixerSoundLevelUpdate(const std::unordered_map<unsigned int, float> &soundLevels){

}

Sound level update callback interface for each single stream in automatic stream mixing onAutoMixerSoundLevelUpdate:

Warning

You can get the streamID in the callback interface only when you have logged in to the room where automatic stream mixing is located and play the mixed stream of this room.

/**
 * Sound level update callback for each single stream in automatic stream mixing
 *
 * Callback notification period is 100 ms.
 * @param soundLevels Sound level key-value pairs for each single stream in stream mixing, key is the streamID of each single stream, value is the sound level value of the corresponding single stream, value ranges from 0.0 to 100.0
 */
public void onAutoMixerSoundLevelUpdate(const std::unordered_map<std::string, float> &soundLevels){

}

/**
 * Sound level update callback for each single stream in automatic stream mixing
 *
 * Callback notification period is 100 ms.
 * @param soundLevels Sound level key-value pairs for each single stream in stream mixing, key is the streamID of each single stream, value is the sound level value of the corresponding single stream, value ranges from 0.0 to 100.0
 */
public void onAutoMixerSoundLevelUpdate(const std::unordered_map<std::string, float> &soundLevels){

}

2 Start Listening to Sound Level Callback Switch

When starting/updating stream mixing, you can start listening to the sound level callback switch.

Manual stream mixing scenario

When calling the startMixerTask interface to initiate a manual stream mixing task, set the enableSoundLevel parameter to true to start listening to sound level, and specify a unique soundLevelID for each input stream:

ZegoMixerTask task;
task.taskID = "task123";
// Enable stream mixing sound level
task.enableSoundLevel = true;

ZegoMixerInput input;
// Assign a soundLevelID to the input stream
input.soundLevelID = 123;

// Other configurations

mSDKEnging->startMixerTask(task, null);

ZegoMixerTask task;
task.taskID = "task123";
// Enable stream mixing sound level
task.enableSoundLevel = true;

ZegoMixerInput input;
// Assign a soundLevelID to the input stream
input.soundLevelID = 123;

// Other configurations

mSDKEnging->startMixerTask(task, null);

Automatic stream mixing scenario

When calling the startAutoMixerTask interface to initiate an automatic stream mixing task, set the enableSoundLevel parameter to true to start listening to sound level:

ZegoAutoMixerTask task;
task.taskID = "autotask123";
// Enable stream mixing sound level
task.enableSoundLevel = true;
// Other configurations

mSDKEnging->startAutoMixerTask(task, null);

ZegoAutoMixerTask task;
task.taskID = "autotask123";
// Enable stream mixing sound level
task.enableSoundLevel = true;
// Other configurations

mSDKEnging->startAutoMixerTask(task, null);

3 Stop Listening to Sound Level Callback Switch

When updating a stream mixing task, you can set the stop listening to sound level callback switch.

Manual stream mixing scenario

When calling the startMixerTask client interface to update a stream mixing task, set the enableSoundLevel parameter to false to stop listening to sound level:

ZegoMixerTask task;
// taskID must remain consistent with the previous one
task.taskID = "task123";
// Stop listening to stream mixing sound level
task.soundLevel = false;

mSDKEnging->startMixerTask(task, null);

ZegoMixerTask task;
// taskID must remain consistent with the previous one
task.taskID = "task123";
// Stop listening to stream mixing sound level
task.soundLevel = false;

mSDKEnging->startMixerTask(task, null);

Automatic stream mixing scenario

When calling the startAutoMixerTask client interface to update an automatic stream mixing task, set the enableSoundLevel parameter to false to stop listening to sound level:

ZegoAutoMixerTask task;
// taskID must remain consistent with the previous one
task.taskID = "autotask123";
// Stop listening to stream mixing sound level
task.enableSoundLevel = false;

mSDKEnging->startAutoMixerTask(task, null);

ZegoAutoMixerTask task;
// taskID must remain consistent with the previous one
task.taskID = "autotask123";
// Stop listening to stream mixing sound level
task.enableSoundLevel = false;

mSDKEnging->startAutoMixerTask(task, null);

FAQ

Why didn't I receive related callbacks after enabling sound level and spectrum monitoring?

The local captured callback will be triggered immediately, and the callback value is 0 when not publishing stream; the remote playing stream callback will only be triggered after successfully playing stream startPlayingStream.