Sound Level and Audio Spectrum

Function Overview

Concept	Description	Application Scenarios	Scenario Diagram
Sound Level	Refers to the volume level of a stream, hereinafter referred to as "sound wave".	During the process of publishing and playing streams, determine which user on the microphone is speaking and display it on the UI.
Audio Spectrum	Refers to the energy value of digital audio signals at each frequency point.	In the host karaoke scenario, under the premise of publishing or playing streams, let the host or audience see animations of pitch and volume changes.

Download Example Source Code

Please refer to Download Example Source Code to get the source code.

For related source code, please view the "Assets/ZegoExpressExample/Examples/AdvancedAudioProcessing/SoundLevelAndAudioSpectrum.cs" file.

Usage Steps for Non-stream Mixing Scenarios

1 Enable sound wave and spectrum monitoring

The SDK disables sound wave and spectrum monitoring by default. Users need to actively call relevant interfaces to enable monitoring. You can start calling interfaces to listen to corresponding callbacks for sound waves or audio spectrum respectively.

Call the StartSoundLevelMonitor interface to start listening to sound waves:

// Start sound wave monitoring
engine.StartSoundLevelMonitor();
// When VAD functionality is needed, use this interface to pass in the corresponding config
engine.StartSoundLevelMonitor(config);

// Start sound wave monitoring
engine.StartSoundLevelMonitor();
// When VAD functionality is needed, use this interface to pass in the corresponding config
engine.StartSoundLevelMonitor(config);

Call the StartAudioSpectrumMonitor interface to start listening to audio spectrum:

// Start audio spectrum monitoring
engine.StartAudioSpectrumMonitor();

// Start audio spectrum monitoring
engine.StartAudioSpectrumMonitor();

2 Listen to sound wave and spectrum callbacks

After enabling sound wave and spectrum monitoring, the SDK will periodically notify users of current sound wave and spectrum data through relevant callbacks (OnCapturedSoundLevelUpdate, OnCapturedSoundLevelInfoUpdate, OnRemoteSoundLevelUpdate, OnRemoteSoundLevelInfoUpdate, OnCapturedAudioSpectrumUpdate, OnRemoteAudioSpectrumUpdate). Users only need to override relevant callback functions to implement UI display.

/**
 * Local captured audio sound wave callback
 * @param soundLevel Local captured sound wave value, ranging from 0.0 to 100.0
 */
void OnCapturedSoundLevelUpdate(float soundLevel) {
    // operate ui in main thread
}

/**
 * Local captured audio sound wave callback, supports human voice detection

 * @note To trigger this callback, you must call the [startSoundLevelMonitor] function to start the sound wave monitor.
 * @note The callback notification period is the parameter value set when calling [startSoundLevelMonitor].
 *
 * @param soundLevelInfo Local captured sound wave value, ranging from 0.0 to 100.0
 */
void OnCapturedSoundLevelInfoUpdate(ZegoSoundLevelInfo soundLevelInfo) {
    // operate ui in main thread
}

/**
 * Remote audio sound wave callback
 * @param soundLevels Key-value pairs of remote sound waves, key is stream ID, value is the sound wave value of the corresponding stream
 */
void OnRemoteSoundLevelUpdate(Dictionary<string, float> soundLevels) {
    // operate ui in main thread
}

/**
 * Remote playing stream audio sound wave callback, supports human voice detection
 *
 * @note To trigger this callback, you must call the [startSoundLevelMonitor] function to start the sound wave monitor, and be in the state of playing stream.
 * @note The callback notification period is the parameter value set when calling [startSoundLevelMonitor].
 *
 * @param soundLevelInfos Key-value pairs of remote sound waves, key is stream ID, value is the sound wave value of the corresponding stream, value ranges from 0.0 to 100.0
 */
void OnRemoteSoundLevelInfoUpdate(Dictionary<string, ZegoSoundLevelInfo> soundLevelInfos) {
    // operate ui in main thread
}

/**
 * Local captured audio spectrum callback
 * @param audioSpectrum Array of local captured audio spectrum values, spectrum value range is [0-2^30]

 */
void OnCapturedAudioSpectrumUpdate(float[] audioSpectrum) {
    // operate ui in main thread
}

/**
 * Remote playing stream audio spectrum callback
 * @param audioSpectrums Key-value pairs of remote audio spectrum, key is stream ID, value is the array of audio spectrum values of the corresponding stream, spectrum value range is [0-2^30]
 */
void OnRemoteAudioSpectrumUpdate(Dictionary<string, float[]> audioSpectrums) {
    // operate ui in main thread
}

/**
 * Local captured audio sound wave callback
 * @param soundLevel Local captured sound wave value, ranging from 0.0 to 100.0
 */
void OnCapturedSoundLevelUpdate(float soundLevel) {
    // operate ui in main thread
}

/**
 * Local captured audio sound wave callback, supports human voice detection

 * @note To trigger this callback, you must call the [startSoundLevelMonitor] function to start the sound wave monitor.
 * @note The callback notification period is the parameter value set when calling [startSoundLevelMonitor].
 *
 * @param soundLevelInfo Local captured sound wave value, ranging from 0.0 to 100.0
 */
void OnCapturedSoundLevelInfoUpdate(ZegoSoundLevelInfo soundLevelInfo) {
    // operate ui in main thread
}

/**
 * Remote audio sound wave callback
 * @param soundLevels Key-value pairs of remote sound waves, key is stream ID, value is the sound wave value of the corresponding stream
 */
void OnRemoteSoundLevelUpdate(Dictionary<string, float> soundLevels) {
    // operate ui in main thread
}

/**
 * Remote playing stream audio sound wave callback, supports human voice detection
 *
 * @note To trigger this callback, you must call the [startSoundLevelMonitor] function to start the sound wave monitor, and be in the state of playing stream.
 * @note The callback notification period is the parameter value set when calling [startSoundLevelMonitor].
 *
 * @param soundLevelInfos Key-value pairs of remote sound waves, key is stream ID, value is the sound wave value of the corresponding stream, value ranges from 0.0 to 100.0
 */
void OnRemoteSoundLevelInfoUpdate(Dictionary<string, ZegoSoundLevelInfo> soundLevelInfos) {
    // operate ui in main thread
}

/**
 * Local captured audio spectrum callback
 * @param audioSpectrum Array of local captured audio spectrum values, spectrum value range is [0-2^30]

 */
void OnCapturedAudioSpectrumUpdate(float[] audioSpectrum) {
    // operate ui in main thread
}

/**
 * Remote playing stream audio spectrum callback
 * @param audioSpectrums Key-value pairs of remote audio spectrum, key is stream ID, value is the array of audio spectrum values of the corresponding stream, spectrum value range is [0-2^30]
 */
void OnRemoteAudioSpectrumUpdate(Dictionary<string, float[]> audioSpectrums) {
    // operate ui in main thread
}

3 Stop sound wave and spectrum monitoring

You can stop calling interfaces to listen to corresponding callbacks for sound waves or audio spectrum respectively.

Call the StopSoundLevelMonitor interface to stop listening to sound waves:

// Stop sound wave monitoring
engine.StopSoundLevelMonitor();

// Stop sound wave monitoring
engine.StopSoundLevelMonitor();

Call the StopAudioSpectrumMonitor interface to stop listening to audio spectrum:

// Stop audio spectrum monitoring
engine.StopAudioSpectrumMonitor();

// Stop audio spectrum monitoring
engine.StopAudioSpectrumMonitor();

Usage Steps for Stream Mixing Scenarios

• Stream mixing is a function that mixes multiple streams into one stream. When customers need to display the sound wave information of each stream before stream mixing, they can use the stream mixing sound wave function. Since the output of stream mixing is a single stream, using the sound wave information of the stream mixing output stream cannot meet the needs of displaying the sound waves of each input stream. At this time, it is necessary to carry the sound wave information of the input stream in the stream information during stream mixing, and then parse the sound wave information of each input stream from the stream information when playing the stream mixing output stream.
• When parsing the sound wave information of each input stream from the stream information, we get the value of the sound wave corresponding to each input stream, which is a dictionary. The key in the dictionary is the identifier of the stream, and the value is the sound wave value. However, due to the size limitation of stream information, key cannot use stream ID, and can only use a numeric ID (soundLevelID) to identify the stream.
• In manual stream mixing configuration, developers need to maintain the association between numeric ID (soundLevelID) and stream ID. In the callback, developers will get the numeric ID (soundLevelID) and corresponding sound wave information.
• In room automatic stream mixing, the stream mixing server and SDK will automatically handle the association between numeric ID and stream ID. In the callback, developers get the sound wave information corresponding to the stream ID.

1 Listen to stream mixing sound wave callback interface

• Interface prototype

  • Sound wave update callback interface for each single stream in manual stream mixing OnMixerSoundLevelUpdate:

    /**
     * Sound wave update notification for each single stream in stream mixing.
     * @param soundLevels Key-value pairs of sound waves of each single stream in stream mixing, key is the soundLevelID of each single stream, value is the sound wave value of the corresponding single stream. Value range: value ranges from 0.0 to 100.0 (this value only represents the sound wave value range of the callback, not the precision).
     */
    public delegate void OnMixerSoundLevelUpdate(Dictionary<uint, float> soundLevels);

    /**
     * Sound wave update notification for each single stream in stream mixing.
     * @param soundLevels Key-value pairs of sound waves of each single stream in stream mixing, key is the soundLevelID of each single stream, value is the sound wave value of the corresponding single stream. Value range: value ranges from 0.0 to 100.0 (this value only represents the sound wave value range of the callback, not the precision).
     */
    public delegate void OnMixerSoundLevelUpdate(Dictionary<uint, float> soundLevels);

  • Sound wave update callback interface for each single stream in automatic stream mixing OnAutoMixerSoundLevelUpdate:

    Warning

    Only when logging into the room where automatic stream mixing is located and playing the stream mixing of this room can you get streamID in the callback interface.

    /**
     * Sound wave update notification for each single stream in automatic stream mixing
     * @param soundLevels Key-value pairs of sound waves of each single stream in automatic stream mixing, key is the streamID of each single stream, value is the sound wave value of the corresponding single stream, value ranges from 0.0 to 100.0 (this value only represents the sound wave value range of the callback, not the precision).
     */
    public delegate void OnAutoMixerSoundLevelUpdate(Dictionary<string, float> soundLevels);

    /**
     * Sound wave update notification for each single stream in automatic stream mixing
     * @param soundLevels Key-value pairs of sound waves of each single stream in automatic stream mixing, key is the streamID of each single stream, value is the sound wave value of the corresponding single stream, value ranges from 0.0 to 100.0 (this value only represents the sound wave value range of the callback, not the precision).
     */
    public delegate void OnAutoMixerSoundLevelUpdate(Dictionary<string, float> soundLevels);

2 Start listening to sound wave callback switch

When starting/updating stream mixing, you can start the switch to listen to sound wave callbacks.

• Manual stream mixing scenario

  When calling the StartMixerTask interface to initiate a manual stream mixing task, set the soundLevel parameter to True to start listening to sound waves, and specify a unique soundLevelID for each input stream:

  ZegoMixerTask task = new ZegoMixerTask();
  task.taskID = "task123";
  // Enable stream mixing sound wave
  task.soundLevel = true;
  
  ZegoMixerInput input = new ZegoMixerInput();
  // Assign a soundLevelID to the input stream
  input.soundLevelID = 123;
  
  // Other configurations
  
  mSDKEnging.StartMixerTask(task, null);

  ZegoMixerTask task = new ZegoMixerTask();
  task.taskID = "task123";
  // Enable stream mixing sound wave
  task.soundLevel = true;
  
  ZegoMixerInput input = new ZegoMixerInput();
  // Assign a soundLevelID to the input stream
  input.soundLevelID = 123;
  
  // Other configurations
  
  mSDKEnging.StartMixerTask(task, null);

• Automatic stream mixing scenario

  When calling the StartAutoMixerTask interface to initiate an automatic stream mixing task, set the enableSoundLevel parameter to True to start listening to sound waves:

  ZegoAutoMixerTask task = new ZegoAutoMixerTask();
  task.taskID = "autotask123";
  // Enable stream mixing sound wave
  task.enableSoundLevel = true;
  // Other configurations
  
  mSDKEnging.StartAutoMixerTask(task, null);

  ZegoAutoMixerTask task = new ZegoAutoMixerTask();
  task.taskID = "autotask123";
  // Enable stream mixing sound wave
  task.enableSoundLevel = true;
  // Other configurations
  
  mSDKEnging.StartAutoMixerTask(task, null);

3 Stop listening to sound wave callback switch

When updating the stream mixing task, you can set to stop the switch to listen to sound wave callbacks.

• Manual stream mixing scenario

  When calling the StartMixerTask client interface to update a stream mixing task, set the soundLevel parameter to false to stop listening to sound waves:

  ZegoMixerTask task = new ZegoMixerTask();
  // taskID must be consistent with the previous one
  task.taskID = "task123";
  // Stop listening to stream mixing sound waves
  task.soundLevel = false;
  
  mSDKEnging.StartMixerTask(task, null);

  ZegoMixerTask task = new ZegoMixerTask();
  // taskID must be consistent with the previous one
  task.taskID = "task123";
  // Stop listening to stream mixing sound waves
  task.soundLevel = false;
  
  mSDKEnging.StartMixerTask(task, null);

• Automatic stream mixing scenario

  When calling the StartAutoMixerTask client interface to update an automatic stream mixing task, set the enableSoundLevel parameter to False to stop listening to sound waves:

  ZegoAutoMixerTask task = new ZegoAutoMixerTask();
  // taskID must be consistent with the previous one
  task.taskID = "autotask123";
  // Stop listening to stream mixing sound waves
  task.enableSoundLevel = false;
  
  mSDKEnging.StartAutoMixerTask(task, null);

  ZegoAutoMixerTask task = new ZegoAutoMixerTask();
  // taskID must be consistent with the previous one
  task.taskID = "autotask123";
  // Stop listening to stream mixing sound waves
  task.enableSoundLevel = false;
  
  mSDKEnging.StartAutoMixerTask(task, null);

FAQ

Why didn't I receive relevant callbacks after enabling the sound wave and spectrum monitoring switch?

The local capture callback will be triggered immediately, and the callback value is 0 when not publishing stream. The remote playing stream callback will only be triggered after successfully playing stream StartPlayingStream .

Related Documentation

• How to handle no audio issue?
• How to handle low volume issue?