AI Noise Reduction

2024-07-27

Feature Overview

ZEGO Express SDK provides AI noise reduction functionality. Developers can perform noise reduction processing on sounds when using a microphone to collect sounds. It synchronously processes non-stationary noise (including mouse click sounds, keyboard sounds, tapping sounds, air conditioner sounds, kitchen dish collision sounds, restaurant noise, environmental wind sounds, cough sounds, blowing sounds, and other non-human vocal noises) while eliminating stationary noise through traditional noise reduction (for details, please refer to Noise Suppression), preserving pure voice and improving users' call experience.

Usage Scenarios

This feature is suitable for voice rooms, conferences, voice chat, and other 1v1 or multi-person audio/video call scenarios.

Eliminable Noise

Developers can use this feature to eliminate the following noise:

Scenario	Some Typical Noises
Meeting Room	Keyboard sounds Table tapping sounds
Office	Keyboard sounds Surrounding colleagues speaking sounds
Vehicle	Siren sounds Car whooshing sounds In-car music sounds Rain sounds and windshield wiper sounds
Internet Cafe	Keyboard sounds Surrounding people's voice sounds
Coffee Shop	Chair dragging sounds Surrounding people's speaking sounds Sharp collision sounds

Effect Demonstration

Office

Original audio includes: mouse click sounds, keyboard sounds, applause sounds, friction sounds, office noise, air conditioner sounds, etc.

After AI noise reduction:

Public Place

Original audio includes: rain sounds, tram sounds, cooking sounds, car whooshing sounds, etc.

After AI noise reduction:

Music Scenario

Original audio:

Conventional AI noise reduction: eliminates noise, but music quality is greatly damaged.

After scenario-based AI noise reduction: eliminates noise, music quality fidelity is preserved.

Prerequisites

Before implementing the AI noise reduction feature, please ensure:

You have created a project in the ZEGOCLOUD Console and applied for a valid AppID and Server address. For details, please refer to Console - Project Information.
You have integrated ZEGO Express SDK in your project and implemented basic audio and video stream publishing and playing functions. For details, please refer to Quick Start - Integration and Quick Start - Implementation.

Usage Steps

Notice

Before calling the enableAiDenoise interface, you need to call the createZegoStream interface first to create a ZegoLocalStream instance object localStream.
Currently only supports AI noise reduction processing for a specific stream, and does not support noise reduction for multiple streams at the same time;
Compatibility:
- PC web: AI noise reduction feature is supported on Google Chrome 66.0, Edge 79.0, Firefox 76.0, and Safari 14.1 and above versions, but to ensure the stability of the AI noise reduction feature, we recommend that you use the latest version of Google Chrome browser or Edge browser.
- Mobile: To ensure the stability of using the AI noise reduction feature, it is recommended to use Safari or Google Chrome 96.0 or above browsers. Some Android WeChat browsers may experience a slight current sound when using the AI noise reduction feature.

Call the createZegoStream interface to create a ZegoLocalStream instance object localStream.

// AI noise reduction module import
import { AiDenoise } from "zego-express-engine-webrtc/aidenoise";
// Needs to be called before new ZegoExpressEngine
ZegoExpressEngine.use(AiDenoise);
// Initialize instance
const zg = new ZegoExpressEngine(appID, server);
// Create media stream
const localStream = await zg.createZegoStream();

// AI noise reduction module import
import { AiDenoise } from "zego-express-engine-webrtc/aidenoise";
// Needs to be called before new ZegoExpressEngine
ZegoExpressEngine.use(AiDenoise);
// Initialize instance
const zg = new ZegoExpressEngine(appID, server);
// Create media stream
const localStream = await zg.createZegoStream();

(Optional) Developers can set the noise reduction mode by calling the setAiDenoiseMode interface.

Notice

The current official website SDK does not include this interface and function. By default, the AI mode is used directly. If you need to use AIBalanced mode, please contact ZEGOCLOUD technical support for a special package.

AI Noise Reduction Mode	Applicable Scenarios
AI	(Default) Lightweight mode, with good noise reduction effects and extremely low performance consumption, suitable for indoor noise environments.
AIBalanced	Balanced mode. Compared with the lightweight mode, under the premise of the same voice fidelity effect, the noise suppression effect is significantly improved, achieving a clean and noise-free or unobtrusive level; but performance consumption is slightly increased. Suitable for outdoor environments with relatively high noise (low SNR) such as streets, roads, markets, etc.

// To use AIBalanced noise reduction mode, you need to remove the import of the AiDenoise module and change to import the VoiceChanger module.
// import { AiDenoise } from "zego-express-engine-webrtc/aidenoise";
// ZegoExpressEngine.use(AiDenoise);

// Audio processing module import
import { VoiceChanger } from "zego-express-engine-webrtc/voice-changer";
// Needs to be called before new ZegoExpressEngine
ZegoExpressEngine.use(VoiceChanger);

// To use AIBalanced noise reduction mode, you need to remove the import of the AiDenoise module and change to import the VoiceChanger module.
// import { AiDenoise } from "zego-express-engine-webrtc/aidenoise";
// ZegoExpressEngine.use(AiDenoise);

// Audio processing module import
import { VoiceChanger } from "zego-express-engine-webrtc/voice-changer";
// Needs to be called before new ZegoExpressEngine
ZegoExpressEngine.use(VoiceChanger);

// Set to AIBalanced
await zg.setAiDenoiseMode(localstream, 1);
// Set to AI
await zg.setAiDenoiseMode(localstream, 0);

// Set to AIBalanced
await zg.setAiDenoiseMode(localstream, 1);
// Set to AI
await zg.setAiDenoiseMode(localstream, 0);

Developers can call the enableAiDenoise interface and pass in the media stream that needs AI noise reduction to enable or disable the AI noise reduction feature, completing the AI noise reduction related settings.
```
// Enable AI noise reduction
await zg.enableAiDenoise(localstream, true);
```
```
// Enable AI noise reduction
await zg.enableAiDenoise(localstream, true);
```