The following is the example code for the business backend that integrates the real-time interactive AI Agent API. You can refer to the example code to implement your own business logic.
Business Backend Example Code
Includes the basic capabilities of obtaining ZEGOCLOUD Token, registering AI Agent, creating AI Agent instances, and deleting AI Agent instances.
Below are the client sample codes, you can refer to these sample codes to implement your own business logic.
Android Client Sample Code
Android client sample code. It includes basic capabilities such as logging into and out of RTC rooms, and publishing and playing streams.
iOS Client Sample Code
iOS client sample code. It includes basic capabilities such as logging into and out of RTC rooms, and publishing and playing streams.
Flutter Client Sample Code
Flutter client sample code. It includes basic capabilities such as logging into and out of RTC rooms, and publishing and playing streams.
Web Client Sample Code
Web client sample code. It includes basic capabilities such as logging into and out of RTC rooms, and publishing and playing streams.
MiniProgram Client Sample Code
MiniProgram client sample code. It includes basic capabilities such as logging into and out of RTC rooms, and publishing and playing streams.
Overall Business Process
Server side: Follow the Server Quick Start guide to run the server sample code and deploy your server
Integrate ZEGOCLOUD AI Agent APIs to manage AI agents.
After completing these two steps, you can add an AI agent to a room for real-time interaction with real users.
Core Capability Implementation
Integrate ZEGO Express SDK
Please refer to Integrate the SDK > 2.2 > Method 2 to manually integrate the SDK. After integrating the SDK, follow these steps to initialize ZegoExpressEngine.
Warning
You must use the ZEGO Express SDK version optimized for AI Agent from the Download SDK and Demo page, otherwise subtitles will not display properly.
1
Add Permission Declaration
Navigate to the "app/src/main" directory, open the "AndroidManifest.xml" file, and add permissions.
private final ActivityResultLauncher<String> requestPermissionLauncher = registerForActivityResult(
new ActivityResultContracts.RequestPermission(), new ActivityResultCallback<Boolean>() {
@Override
public void onActivityResult(Boolean isGranted) {
if (isGranted) {
// Permission granted
}
}
});
// Initiate request
requestPermissionLauncher.launch(Manifest.permission.RECORD_AUDIO);
private final ActivityResultLauncher<String> requestPermissionLauncher = registerForActivityResult(
new ActivityResultContracts.RequestPermission(), new ActivityResultCallback<Boolean>() {
@Override
public void onActivityResult(Boolean isGranted) {
if (isGranted) {
// Permission granted
}
}
});
// Initiate request
requestPermissionLauncher.launch(Manifest.permission.RECORD_AUDIO);
3
Create and Initialize ZegoExpressEngine
ZegoEngineProfile zegoEngineProfile = new ZegoEngineProfile();
zegoEngineProfile.appID = ; // Obtain from ZEGOCLOUD Console
// !mark
zegoEngineProfile.scenario = ZegoScenario.HIGH_QUALITY_CHATROOM; // Setting this scenario can avoid requesting camera permissions, and the integrator should set specific values according to their own business scenarios
zegoEngineProfile.application = getApplication();
ZegoExpressEngine.createEngine(zegoEngineProfile, null);
ZegoEngineProfile zegoEngineProfile = new ZegoEngineProfile();
zegoEngineProfile.appID = ; // Obtain from ZEGOCLOUD Console
// !mark
zegoEngineProfile.scenario = ZegoScenario.HIGH_QUALITY_CHATROOM; // Setting this scenario can avoid requesting camera permissions, and the integrator should set specific values according to their own business scenarios
zegoEngineProfile.application = getApplication();
ZegoExpressEngine.createEngine(zegoEngineProfile, null);
Please refer to Import the SDK > 2.2 > Method 3 to manually integrate the SDK. After integrating the SDK, follow these steps to initialize ZegoExpressEngine.
Warning
You must use the ZEGO Express SDK version optimized for AI Agent from the Download SDK and Demo page, otherwise subtitles will not display properly.
- (void)requestAudioPermission:(void(^)(BOOL granted))completion {
/// Need to add a description of microphone usage in the project's Info.plist file
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
[audioSession requestRecordPermission:^(BOOL granted) {
dispatch_async(dispatch_get_main_queue(), ^{
completion(granted);
});
}];
}
- (void)requestAudioPermission:(void(^)(BOOL granted))completion {
/// Need to add a description of microphone usage in the project's Info.plist file
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
[audioSession requestRecordPermission:^(BOOL granted) {
dispatch_async(dispatch_get_main_queue(), ^{
completion(granted);
});
}];
}
3
Create and Initialize ZegoExpressEngine
-(void)initZegoExpressEngine{
ZegoEngineProfile* profile = [[ZegoEngineProfile alloc]init];
profile.appID = kZegoPassAppId;
// !mark
profile.scenario = ZegoScenarioHighQualityChatroom; // Setting this scenario can avoid requesting camera permissions, and the integrator should set specific values according to their own business scenarios
[ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
}
-(void)initZegoExpressEngine{
ZegoEngineProfile* profile = [[ZegoEngineProfile alloc]init];
profile.appID = kZegoPassAppId;
// !mark
profile.scenario = ZegoScenarioHighQualityChatroom; // Setting this scenario can avoid requesting camera permissions, and the integrator should set specific values according to their own business scenarios
[ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
}
Please refer to Integrate the SDK > Method 1 to integrate the SDK using "pub" dependency. After integrating the SDK, follow these steps to initialize ZegoExpressEngine.
Go to ios/Runner directory, open Info.plist file, and add permissions
Info.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
...
<key>UIBackgroundModes</key>
<array>
<string>audio</string>
</array>
<key>NSMicrophoneUsageDescription</key>
<string>Need to access microphone to chat</string>
</dict>
</plist>
Info.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
...
<key>UIBackgroundModes</key>
<array>
<string>audio</string>
</array>
<key>NSMicrophoneUsageDescription</key>
<string>Need to access microphone to chat</string>
</dict>
</plist>
3
Go to ios directory, open Podfile file, and add permissions
Podfile
post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
<!-- !mark(1:8) -->
# Start of the permission_handler configuration
target.build_configurations.each do |config|
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
'$(inherited)',
'PERMISSION_MICROPHONE=1',
]
end
# End of the permission_handler configuration
end
end
Podfile
post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
<!-- !mark(1:8) -->
# Start of the permission_handler configuration
target.build_configurations.each do |config|
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
'$(inherited)',
'PERMISSION_MICROPHONE=1',
]
end
# End of the permission_handler configuration
end
end
await ZegoExpressEngine.createEngineWithProfile(
/// Set this scenario to avoid requesting camera permissions, and the integrator should set specific values according to their own business scenarios
// !mark
ZegoEngineProfile(ZegoKey.appId, ZegoScenario.HighQualityChatroom),
);
await ZegoExpressEngine.createEngineWithProfile(
/// Set this scenario to avoid requesting camera permissions, and the integrator should set specific values according to their own business scenarios
// !mark
ZegoEngineProfile(ZegoKey.appId, ZegoScenario.HighQualityChatroom),
);
Please refer to Integrate SDK > Method 2 to use npm to integrate the ZEGO Express SDK version optimized for AI Agent. After integrating the SDK, initialize ZegoExpressEngine as follows.
Warning
You must use the ZEGO Express SDK version optimized for AI Agent from the Download SDK and Demo page, otherwise subtitles will not display properly.
Instantiate ZegoExpressEngine
Check system requirements (WebRTC support and microphone permissions)
import { ZegoExpressEngine } from "zego-express-engine-webrtc";
const appID = 1234567 // Obtain from ZEGOCLOUD Console
const server = 'xxx' // Obtain from ZEGOCLOUD Console
// Instantiate ZegoExpressEngine with appId and server configurations
// !mark
const zg = new ZegoExpressEngine(appID, server);
// Check system requirements
// !mark
const checkSystemRequirements = async () => {
// Detect WebRTC support
const rtc_sup = await zg.checkSystemRequirements("webRTC");
if (!rtc_sup.result) {
// Browser does not support WebRTC
}
// Detect microphone permission status
const mic_sup = await zg.checkSystemRequirements("microphone");
if (!mic_sup.result) {
// Microphone permission is not enabled
}
}
checkSystemRequirements()
import { ZegoExpressEngine } from "zego-express-engine-webrtc";
const appID = 1234567 // Obtain from ZEGOCLOUD Console
const server = 'xxx' // Obtain from ZEGOCLOUD Console
// Instantiate ZegoExpressEngine with appId and server configurations
// !mark
const zg = new ZegoExpressEngine(appID, server);
// Check system requirements
// !mark
const checkSystemRequirements = async () => {
// Detect WebRTC support
const rtc_sup = await zg.checkSystemRequirements("webRTC");
if (!rtc_sup.result) {
// Browser does not support WebRTC
}
// Detect microphone permission status
const mic_sup = await zg.checkSystemRequirements("microphone");
if (!mic_sup.result) {
// Microphone permission is not enabled
}
}
checkSystemRequirements()
Please refer to Integrate SDK to integrate the SDK. After integrating the SDK, initialize ZegoExpressEngine as follows.
Instantiate ZegoExpressEngine
Check system requirements (WebRTC support and microphone permissions)
import { ZegoExpressEngine } from "zego-express-engine-miniprogram";
const appID = 1234567 // Obtain from ZEGOCLOUD Console
const server = 'xxx' // Obtain from ZEGOCLOUD Console
// Instantiate ZegoExpressEngine with appId and server configurations
// !mark
const zg = new ZegoExpressEngine(appID, server);
// Initialize mini program component instance
zg.initContext({
wxContext: context,
pushAtr: "pusher", // The variable name configured for pushAtr must be exactly the same as the variable name passed to the pusher attribute of the <zego-pusher> component.
playAtr: "playerList" // The variable name configured for playAtr must be exactly the same as the variable name passed to the playerList attribute of the <zego-player> component.
})
// Check system requirements
// !mark
const checkSystemRequirements = async () => {
const rtc_sup = await zg.checkSystemRequirements();
if (res.code === 10002) {
// Microphone permission not enabled
}
if (res.code === 10001) {
// WeChat version too low to use related components
}
}
checkSystemRequirements()
import { ZegoExpressEngine } from "zego-express-engine-miniprogram";
const appID = 1234567 // Obtain from ZEGOCLOUD Console
const server = 'xxx' // Obtain from ZEGOCLOUD Console
// Instantiate ZegoExpressEngine with appId and server configurations
// !mark
const zg = new ZegoExpressEngine(appID, server);
// Initialize mini program component instance
zg.initContext({
wxContext: context,
pushAtr: "pusher", // The variable name configured for pushAtr must be exactly the same as the variable name passed to the pusher attribute of the <zego-pusher> component.
playAtr: "playerList" // The variable name configured for playAtr must be exactly the same as the variable name passed to the playerList attribute of the <zego-player> component.
})
// Check system requirements
// !mark
const checkSystemRequirements = async () => {
const rtc_sup = await zg.checkSystemRequirements();
if (res.code === 10002) {
// Microphone permission not enabled
}
if (res.code === 10001) {
// WeChat version too low to use related components
}
}
checkSystemRequirements()
Create WXML for Corresponding Business Scenario
Copy component code to project
Copy the zego-player and zego-pusher folders from the components folder of the sample code to your business code's components folder.
Import components in project JSON file
According to your project structure, import the <zego-pusher> and <zego-player> components in the corresponding JSON file.
Use the push/pull stream components <zego-pusher> and <zego-player> in the WXML file.
// Use components in WXML file
// The variable name passed to the pusher attribute of the <zego-pusher> component must be exactly the same as the parameter value of pushAtr in the initContext configuration.
<zego-pusher id="zegoPusher" pusher="{{pusher}}" />
// The variable name passed to the playerList attribute of the <zego-player> component must be exactly the same as the parameter value of playAtr in the initContext configuration. zegoPlayerList is described in the "Pull other users' audio" section.
<zego-player wx:for="{{zegoPlayerList}}" wx:key="id" id="{{item.componentID}}" playerId="{{item.playerId}}" playerList="{{playerList}}" />
// Use components in WXML file
// The variable name passed to the pusher attribute of the <zego-pusher> component must be exactly the same as the parameter value of pushAtr in the initContext configuration.
<zego-pusher id="zegoPusher" pusher="{{pusher}}" />
// The variable name passed to the playerList attribute of the <zego-player> component must be exactly the same as the parameter value of playAtr in the initContext configuration. zegoPlayerList is described in the "Pull other users' audio" section.
<zego-player wx:for="{{zegoPlayerList}}" wx:key="id" id="{{item.componentID}}" playerId="{{item.playerId}}" playerList="{{playerList}}" />
Notify Your Server to Start Call
You can notify your server to start the call immediately after the real user enters the room on the client side. Asynchronous calls can help reduce call connection time. After receiving the start call notification, your server creates an AI agent instance using the same roomID and associated userID and streamID create an AI agent instance , so that the AI agent can interact with real users in the same room through mutual stream publishing and playing.
Note
By default, each account can have at most 10 AI agent instances. If the limit is exceeded, the creation of an AI agent instance will fail. If you need to adjust this limit, please contact ZEGOCLOUD Technical Support.
Sample Code for Notifying Your Server
Note
In the following examples, roomID, userID, streamID and other parameters are not passed when notifying your server to start the call because fixed values have been agreed between the client and your server in this example. In actual use, please pass the real parameters according to your business requirements.
// Notify your server to start call
private void start() {
RequestBody body = RequestBody.create("", MediaType.parse("application/json; charset=utf-8"));
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
Request request = new Request.Builder().url(YOUR_SERVER_URL + "/api/start").post(body).build();
new OkHttpClient.Builder().build().newCall(request).enqueue(new Callback() {
@Override
public void onFailure(@NonNull Call call, @NonNull IOException e) {
}
@Override
public void onResponse(@NonNull Call call, @NonNull Response response) throws IOException {
}
});
}
// Notify your server to start call
private void start() {
RequestBody body = RequestBody.create("", MediaType.parse("application/json; charset=utf-8"));
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
Request request = new Request.Builder().url(YOUR_SERVER_URL + "/api/start").post(body).build();
new OkHttpClient.Builder().build().newCall(request).enqueue(new Callback() {
@Override
public void onFailure(@NonNull Call call, @NonNull IOException e) {
}
@Override
public void onResponse(@NonNull Call call, @NonNull Response response) throws IOException {
}
});
}
// Notify your server to start call
/**
* Start a call with the AI agent
*
* @param completion Completion callback, returns operation result
* @discussion This method sends a request to the server to start the call, used to initialize the AI agent instance
*/
- (void)doStartCallWithCompletion:(void (^)(NSInteger code, NSString *message, NSDictionary *data))completion {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
NSString *url = [NSString stringWithFormat:@"%@/api/start", self.currentBaseURL];
NSURL *requestURL = [NSURL URLWithString:url];
// Create request
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:requestURL];
request.HTTPMethod = @"POST";
// Set request headers
[request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
// Create request parameters
NSMutableDictionary *params = [NSMutableDictionary dictionary];
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:params options:0 error:nil];
request.HTTPBody = jsonData;
// Create session
NSURLSession *session = [NSURLSession sharedSession];
// Send request
NSURLSessionDataTask *task = [session dataTaskWithRequest:request
completionHandler:^(NSData * _Nullable data,
NSURLResponse * _Nullable response,
NSError * _Nullable error) {
dispatch_async(dispatch_get_main_queue(), ^{
if (error) {
if (completion) {
completion(-1, @"Network request failed", nil);
}
return;
}
NSHTTPURLResponse *httpUrlResponse = (NSHTTPURLResponse *)response;
if (httpUrlResponse.statusCode != 200) {
if (completion) {
completion(httpUrlResponse.statusCode,
[NSString stringWithFormat:@"Server error: %ld", (long)httpUrlResponse.statusCode],
nil);
}
return;
}
NSError *jsonError;
NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data options:0 error:&jsonError];
if (jsonError) {
if (completion) {
completion(-2, @"Failed to parse response data", nil);
}
return;
}
// Parse response data
NSInteger code = [dict[@"code"] integerValue];
NSString *message = dict[@"message"];
NSDictionary *responseData = dict[@"data"];
if (completion) {
completion(code, message, responseData);
}
});
}];
[task resume];
}
// Notify your server to start call
/**
* Start a call with the AI agent
*
* @param completion Completion callback, returns operation result
* @discussion This method sends a request to the server to start the call, used to initialize the AI agent instance
*/
- (void)doStartCallWithCompletion:(void (^)(NSInteger code, NSString *message, NSDictionary *data))completion {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
NSString *url = [NSString stringWithFormat:@"%@/api/start", self.currentBaseURL];
NSURL *requestURL = [NSURL URLWithString:url];
// Create request
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:requestURL];
request.HTTPMethod = @"POST";
// Set request headers
[request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
// Create request parameters
NSMutableDictionary *params = [NSMutableDictionary dictionary];
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:params options:0 error:nil];
request.HTTPBody = jsonData;
// Create session
NSURLSession *session = [NSURLSession sharedSession];
// Send request
NSURLSessionDataTask *task = [session dataTaskWithRequest:request
completionHandler:^(NSData * _Nullable data,
NSURLResponse * _Nullable response,
NSError * _Nullable error) {
dispatch_async(dispatch_get_main_queue(), ^{
if (error) {
if (completion) {
completion(-1, @"Network request failed", nil);
}
return;
}
NSHTTPURLResponse *httpUrlResponse = (NSHTTPURLResponse *)response;
if (httpUrlResponse.statusCode != 200) {
if (completion) {
completion(httpUrlResponse.statusCode,
[NSString stringWithFormat:@"Server error: %ld", (long)httpUrlResponse.statusCode],
nil);
}
return;
}
NSError *jsonError;
NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data options:0 error:&jsonError];
if (jsonError) {
if (completion) {
completion(-2, @"Failed to parse response data", nil);
}
return;
}
// Parse response data
NSInteger code = [dict[@"code"] integerValue];
NSString *message = dict[@"message"];
NSDictionary *responseData = dict[@"data"];
if (completion) {
completion(code, message, responseData);
}
});
}];
[task resume];
}
// Notify your server to start call
Future<Map<String, dynamic>> startCall() async {
try {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark(2:2)
final response = await http.post(
Uri.parse('$_currentBaseUrl/api/start'),
headers: {'Content-Type': 'application/json'},
);
if (response.statusCode == 200) {
final json = jsonDecode(response.body);
return json;
}
return {'code': -1, 'message': 'Request failed'};
} catch (e) {
return {'code': -1, 'message': e.toString()};
}
}
// Notify your server to start call
Future<Map<String, dynamic>> startCall() async {
try {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark(2:2)
final response = await http.post(
Uri.parse('$_currentBaseUrl/api/start'),
headers: {'Content-Type': 'application/json'},
);
if (response.statusCode == 200) {
final json = jsonDecode(response.body);
return json;
}
return {'code': -1, 'message': 'Request failed'};
} catch (e) {
return {'code': -1, 'message': e.toString()};
}
}
// Notify your server to start call
async function startCall() {
try {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
const response = await fetch(`${YOUR_SERVER_URL}/api/start`, { // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
});
const data = await response.json();
console.log('Start call result:', data);
return data;
} catch (error) {
console.error('Failed to start call:', error);
throw error;
}
}
// Notify your server to start call
async function startCall() {
try {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
const response = await fetch(`${YOUR_SERVER_URL}/api/start`, { // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
});
const data = await response.json();
console.log('Start call result:', data);
return data;
} catch (error) {
console.error('Failed to start call:', error);
throw error;
}
}
// Notify your server to start call
async function startCall() {
try {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
const response = await wx.request({
url: `${YOUR_SERVER_URL}/api/start`, // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
header: {
'Content-Type': 'application/json',
}
});
console.log('Start call result:', response.data);
return response.data;
} catch (error) {
console.error('Failed to start call:', error);
throw error;
}
}
// Notify your server to start call
async function startCall() {
try {
// The server calls the createAgentInstance interface in the /api/startn interface
// !mark
const response = await wx.request({
url: `${YOUR_SERVER_URL}/api/start`, // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
header: {
'Content-Type': 'application/json',
}
});
console.log('Start call result:', response.data);
return response.data;
} catch (error) {
console.error('Failed to start call:', error);
throw error;
}
}
User logs in a RTC room and starts publishing a stream
After a real user logs into the room, they start publishing streams.
Note
In this scenario, AI echo cancellation should be enabled for better effects.
The token used for login needs to be obtained from your server; please refer to the complete sample code.
Note
Please ensure that the roomID, userID, and streamID are unique under one ZEGOCLOUD APPID.
roomID: Generated by the user according to their own rules, it will be used to log into the Express SDK room. Only numbers, English characters, and '~', '!', '@', '#', '$', '%', '^', '&', '*', '(', ')', '_', '+', '=', '-', '`', ';', ''', ',', '.', '<', '>', '' are supported. If interoperability with the Web SDK is required, do not use '%'.
userID: Length should not exceed 32 bytes. Only numbers, English characters, and '~', '!', '@', '#', '$', '%', '^', '&', '*', '(', ')', '_', '+', '=', '-', '`', ';', ''', ',', '.', '<', '>', '' are supported. If interoperability with the Web SDK is required, do not use '%'.
streamID: Length should not exceed 256 bytes. Only numbers, English characters, and '-', '_' are supported.
Client login to room and publish a stream
private void loginRoom(String userId, String userName, String userName, String token,
IZegoRoomLoginCallback callback) {
ZegoEngineConfig config = new ZegoEngineConfig();
HashMap<String, String> advanceConfig = new HashMap<String, String>();
// !mark(1:2)
advanceConfig.put("set_audio_volume_ducking_mode", "1");
advanceConfig.put("enable_rnd_volume_adaptive", "true");
config.advancedConfig = advanceConfig;
ZegoExpressEngine.setEngineConfig(config);
ZegoExpressEngine.getEngine().setRoomScenario(ZegoScenario.HIGH_QUALITY_CHATROOM);
ZegoExpressEngine.getEngine().setAudioDeviceMode(ZegoAudioDeviceMode.GENERAL);
// !mark(1:6)
ZegoExpressEngine.getEngine().enableAEC(true);
// Enable AI echo cancellation
ZegoExpressEngine.getEngine().setAECMode(ZegoAECMode.AI_BALANCED);
ZegoExpressEngine.getEngine().enableAGC(true);
ZegoExpressEngine.getEngine().enableANS(true);
ZegoExpressEngine.getEngine().setANSMode(ZegoANSMode.MEDIUM);
ZegoRoomConfig roomConfig = new ZegoRoomConfig();
roomConfig.isUserStatusNotify = true;
roomConfig.token = token; // Token authentication is required, obtain it from your server, and refer to ZEGOCLOUD documentation for generation method
String roomId ; // Custom room ID for login, please refer to the format description
String userSteamID // Custom stream ID for publishing, please refer to the format description
// !mark
ZegoExpressEngine.getEngine()
.loginRoom(roomId, new ZegoUser(userId, userName), roomConfig, (errorCode, extendedData) -> {
Timber.d(
"loginRoom() called with: errorCode = [" + errorCode + "], extendedData = [" + extendedData + "]");
if (errorCode == 0) {
// !mark
// Start publishing stream after successful login
ZegoExpressEngine.getEngine().startPublishingStream(userSteamID);
// Set microphone mute status, false means unmuted, true means muted
ZegoExpressEngine.getEngine().muteMicrophone(false);
}
if (callback != null) {
callback.onRoomLoginResult(errorCode, extendedData);
}
});
}
Client login to room and publish a stream
private void loginRoom(String userId, String userName, String userName, String token,
IZegoRoomLoginCallback callback) {
ZegoEngineConfig config = new ZegoEngineConfig();
HashMap<String, String> advanceConfig = new HashMap<String, String>();
// !mark(1:2)
advanceConfig.put("set_audio_volume_ducking_mode", "1");
advanceConfig.put("enable_rnd_volume_adaptive", "true");
config.advancedConfig = advanceConfig;
ZegoExpressEngine.setEngineConfig(config);
ZegoExpressEngine.getEngine().setRoomScenario(ZegoScenario.HIGH_QUALITY_CHATROOM);
ZegoExpressEngine.getEngine().setAudioDeviceMode(ZegoAudioDeviceMode.GENERAL);
// !mark(1:6)
ZegoExpressEngine.getEngine().enableAEC(true);
// Enable AI echo cancellation
ZegoExpressEngine.getEngine().setAECMode(ZegoAECMode.AI_BALANCED);
ZegoExpressEngine.getEngine().enableAGC(true);
ZegoExpressEngine.getEngine().enableANS(true);
ZegoExpressEngine.getEngine().setANSMode(ZegoANSMode.MEDIUM);
ZegoRoomConfig roomConfig = new ZegoRoomConfig();
roomConfig.isUserStatusNotify = true;
roomConfig.token = token; // Token authentication is required, obtain it from your server, and refer to ZEGOCLOUD documentation for generation method
String roomId ; // Custom room ID for login, please refer to the format description
String userSteamID // Custom stream ID for publishing, please refer to the format description
// !mark
ZegoExpressEngine.getEngine()
.loginRoom(roomId, new ZegoUser(userId, userName), roomConfig, (errorCode, extendedData) -> {
Timber.d(
"loginRoom() called with: errorCode = [" + errorCode + "], extendedData = [" + extendedData + "]");
if (errorCode == 0) {
// !mark
// Start publishing stream after successful login
ZegoExpressEngine.getEngine().startPublishingStream(userSteamID);
// Set microphone mute status, false means unmuted, true means muted
ZegoExpressEngine.getEngine().muteMicrophone(false);
}
if (callback != null) {
callback.onRoomLoginResult(errorCode, extendedData);
}
});
}
Client request to login to room and publish a stream
// Record the agent
self.streamToPlay = [self getAgentStreamID];
ZegoEngineConfig* engineConfig = [[ZegoEngineConfig alloc] init];
engineConfig.advancedConfig = @{
// !mark(1:2)
@"set_audio_volume_ducking_mode":@1,/** This configuration is used for volume ducking **/
@"enable_rnd_volume_adaptive":@"true",/** This configuration is used for adaptive playback volume **/
};
// !mark
[ZegoExpressEngine setEngineConfig:engineConfig];
// This setting only affects AEC (echo cancellation). Here we set it to ModeGeneral, which uses our proprietary echo cancellation algorithm, giving us more control.
// If other options are selected, it might use the system's echo cancellation, which may work better on iPhones but could be less effective on some Android devices.
[[ZegoExpressEngine sharedEngine] setAudioDeviceMode:ZegoAudioDeviceModeGeneral];
// Enable AI echo cancellation
// !mark(1:5)
[[ZegoExpressEngine sharedEngine] enableAGC:TRUE];
[[ZegoExpressEngine sharedEngine] enableAEC:TRUE];
[[ZegoExpressEngine sharedEngine] setAECMode:ZegoAECModeAIBalanced];
[[ZegoExpressEngine sharedEngine] enableANS:TRUE];
[[ZegoExpressEngine sharedEngine] setANSMode:ZegoANSModeMedium];
// Login to room
// !mark
[self loginRoom:^(int errorCode, NSDictionary *extendedData) {
if (errorCode!=0) {
NSString* errorMsg =[NSString stringWithFormat:@"Failed to enter voice room:%d", errorCode];
completion(NO, errorMsg);
return;
}
// Start publishing stream after entering room
[self startPushlishStream];
}];
Client request to login to room and publish a stream
// Record the agent
self.streamToPlay = [self getAgentStreamID];
ZegoEngineConfig* engineConfig = [[ZegoEngineConfig alloc] init];
engineConfig.advancedConfig = @{
// !mark(1:2)
@"set_audio_volume_ducking_mode":@1,/** This configuration is used for volume ducking **/
@"enable_rnd_volume_adaptive":@"true",/** This configuration is used for adaptive playback volume **/
};
// !mark
[ZegoExpressEngine setEngineConfig:engineConfig];
// This setting only affects AEC (echo cancellation). Here we set it to ModeGeneral, which uses our proprietary echo cancellation algorithm, giving us more control.
// If other options are selected, it might use the system's echo cancellation, which may work better on iPhones but could be less effective on some Android devices.
[[ZegoExpressEngine sharedEngine] setAudioDeviceMode:ZegoAudioDeviceModeGeneral];
// Enable AI echo cancellation
// !mark(1:5)
[[ZegoExpressEngine sharedEngine] enableAGC:TRUE];
[[ZegoExpressEngine sharedEngine] enableAEC:TRUE];
[[ZegoExpressEngine sharedEngine] setAECMode:ZegoAECModeAIBalanced];
[[ZegoExpressEngine sharedEngine] enableANS:TRUE];
[[ZegoExpressEngine sharedEngine] setANSMode:ZegoANSModeMedium];
// Login to room
// !mark
[self loginRoom:^(int errorCode, NSDictionary *extendedData) {
if (errorCode!=0) {
NSString* errorMsg =[NSString stringWithFormat:@"Failed to enter voice room:%d", errorCode];
completion(NO, errorMsg);
return;
}
// Start publishing stream after entering room
[self startPushlishStream];
}];
Client request to login to room and publish a stream
final String _userId = 'user_id_1';
final String _roomId = 'room_id_1';
final String _userStreamId = 'user_stream_id_1';
/// Generate RTC Token [Reference Documentation]（https://www.zegocloud.com/docs/video-call/token?platform=flutter&language=dart）
final token = await getToken();
if (token.isEmpty) {
return false;
}
// !mark(1:11)
/// The following is used for answering delay optimization, you need to integrate the corresponding version of ZegoExpressEngine sdk, please contact ZEGOCLOUD technical support
ZegoExpressEngine.setEngineConfig(
ZegoEngineConfig(
advancedConfig: {
/**This configuration is used for volume ducking**/
'set_audio_volume_ducking_mode': '1',
/**This configuration is used for adaptive playback volume**/
'enable_rnd_volume_adaptive': 'true'
},
),
);
/// Enable 3A
// !mark(1:3)
ZegoExpressEngine.instance.enableAGC(true);
ZegoExpressEngine.instance.enableAEC(true);
if (!kIsWeb) {
ZegoExpressEngine.instance.setAECMode(ZegoANSMode.AIAGGRESSIVE2);
// !mark(1:4)
/// This setting only affects AEC (echo cancellation). Here we set it to ModeGeneral, which uses our proprietary echo cancellation, which is more controllable.
/// If other options are selected, it might use the system's echo cancellation, which may work better on iPhones but could be less effective on some Android devices.
ZegoExpressEngine.instance.setAudioDeviceMode(
ZegoAudioDeviceMode.General,
);
}
ZegoExpressEngine.instance.enableANS(true);
ZegoExpressEngine.instance.setANSMode(ZegoANSMode.Medium);
/// Login to room
// !mark
final user = ZegoUser(_userId, _userId);
final roomConfig = ZegoRoomConfig.defaultConfig()
..isUserStatusNotify = true
..token = token;
final loginResult = await ZegoExpressEngine.instance.loginRoom(
_roomId,
user,
config: roomConfig,
);
if (0 != loginResult.errorCode && 1002001 != loginResult.errorCode) {
return false;
}
/// Start publishing stream (open microphone)
await ZegoExpressEngine.instance.muteMicrophone(false);
await ZegoExpressEngine.instance.startPublishingStream(_userStreamId);
Client request to login to room and publish a stream
final String _userId = 'user_id_1';
final String _roomId = 'room_id_1';
final String _userStreamId = 'user_stream_id_1';
/// Generate RTC Token [Reference Documentation]（https://www.zegocloud.com/docs/video-call/token?platform=flutter&language=dart）
final token = await getToken();
if (token.isEmpty) {
return false;
}
// !mark(1:11)
/// The following is used for answering delay optimization, you need to integrate the corresponding version of ZegoExpressEngine sdk, please contact ZEGOCLOUD technical support
ZegoExpressEngine.setEngineConfig(
ZegoEngineConfig(
advancedConfig: {
/**This configuration is used for volume ducking**/
'set_audio_volume_ducking_mode': '1',
/**This configuration is used for adaptive playback volume**/
'enable_rnd_volume_adaptive': 'true'
},
),
);
/// Enable 3A
// !mark(1:3)
ZegoExpressEngine.instance.enableAGC(true);
ZegoExpressEngine.instance.enableAEC(true);
if (!kIsWeb) {
ZegoExpressEngine.instance.setAECMode(ZegoANSMode.AIAGGRESSIVE2);
// !mark(1:4)
/// This setting only affects AEC (echo cancellation). Here we set it to ModeGeneral, which uses our proprietary echo cancellation, which is more controllable.
/// If other options are selected, it might use the system's echo cancellation, which may work better on iPhones but could be less effective on some Android devices.
ZegoExpressEngine.instance.setAudioDeviceMode(
ZegoAudioDeviceMode.General,
);
}
ZegoExpressEngine.instance.enableANS(true);
ZegoExpressEngine.instance.setANSMode(ZegoANSMode.Medium);
/// Login to room
// !mark
final user = ZegoUser(_userId, _userId);
final roomConfig = ZegoRoomConfig.defaultConfig()
..isUserStatusNotify = true
..token = token;
final loginResult = await ZegoExpressEngine.instance.loginRoom(
_roomId,
user,
config: roomConfig,
);
if (0 != loginResult.errorCode && 1002001 != loginResult.errorCode) {
return false;
}
/// Start publishing stream (open microphone)
await ZegoExpressEngine.instance.muteMicrophone(false);
await ZegoExpressEngine.instance.startPublishingStream(_userStreamId);
Client login to room and publish a stream
const userId = "" // User ID for logging into the Express SDK room
const roomId = "" // RTC Room ID
const userStreamId = "" // User stream push ID
async function enterRoom() {
try {
// Generate RTC Token [Reference Documentation] (https://www.zegocloud.com/docs/video-call/token?platform=web&language=javascript)
const token = await Api.getToken();
// Login to room
await zg.loginRoom(roomId, token, {
userID: userId,
userName: "",
});
// Create local audio stream
const localStream = await zg.createZegoStream({
camera: {
video: false,
audio: true,
},
});
if (localStream) {
// !mark(1:2)
// Push local stream
await zg.startPublishingStream(userStreamId, localStream);
}
} catch (error) {
console.error("Failed to enter room:", error);
throw error;
}
}
enterRoom()
Client login to room and publish a stream
const userId = "" // User ID for logging into the Express SDK room
const roomId = "" // RTC Room ID
const userStreamId = "" // User stream push ID
async function enterRoom() {
try {
// Generate RTC Token [Reference Documentation] (https://www.zegocloud.com/docs/video-call/token?platform=web&language=javascript)
const token = await Api.getToken();
// Login to room
await zg.loginRoom(roomId, token, {
userID: userId,
userName: "",
});
// Create local audio stream
const localStream = await zg.createZegoStream({
camera: {
video: false,
audio: true,
},
});
if (localStream) {
// !mark(1:2)
// Push local stream
await zg.startPublishingStream(userStreamId, localStream);
}
} catch (error) {
console.error("Failed to enter room:", error);
throw error;
}
}
enterRoom()
Client login to room and publish a stream
const userId = "" // User ID for logging into the ZEGO Express SDK room
const roomId = "" // RTC Room ID
const userStreamId = "" // User stream push ID
async function enterRoom() {
try {
// Generate RTC Token [Reference Documentation] (/real-time-video-web/communication/using-token-authentication)
const token = await Api.getToken();
// Login to room
await zg.loginRoom(roomId, token, {
userID: userId,
userName: "",
});
// Create local audio stream
const zegoPusher = this.selectComponent("#zegoPusher");
// !mark(1:2)
// Push local stream
await zegoPusher.startPush(zg, userStreamId, {}, {enableCamera: false, mode: 'RTC'});
} catch (error) {
console.error("Failed to enter room:", error);
throw error;
}
}
enterRoom()
Client login to room and publish a stream
const userId = "" // User ID for logging into the ZEGO Express SDK room
const roomId = "" // RTC Room ID
const userStreamId = "" // User stream push ID
async function enterRoom() {
try {
// Generate RTC Token [Reference Documentation] (/real-time-video-web/communication/using-token-authentication)
const token = await Api.getToken();
// Login to room
await zg.loginRoom(roomId, token, {
userID: userId,
userName: "",
});
// Create local audio stream
const zegoPusher = this.selectComponent("#zegoPusher");
// !mark(1:2)
// Push local stream
await zegoPusher.startPush(zg, userStreamId, {}, {enableCamera: false, mode: 'RTC'});
} catch (error) {
console.error("Failed to enter room:", error);
throw error;
}
}
enterRoom()
Play the AI Agent Stream
By default, there is only one real user and one AI agent in the same room, so any new stream added is assumed to be the AI agent stream.
Client request to play the AI agent stream
// Set up the event handler
void setEventHandler() {
ZegoExpressEngine.getEngine().setEventHandler(new IZegoEventHandler() {
@Override
// When other users in the room start/stop publishing streams, you can receive notifications about the corresponding user's audio/video stream changes here
public void onRoomStreamUpdate(String roomID, ZegoUpdateType updateType, ArrayList<ZegoStream> streamList, JSONObject extendedData) {
super.onRoomStreamUpdate(roomID, updateType, streamList, extendedData);
// When updateType is ZegoUpdateType.ADD, it means there is a new audio/video stream, at this time we can call the startPlayingStream interface to pull this audio/video stream
// !mark(1:8)
if (updateType == ZegoUpdateType.ADD) {
ZegoStream stream = streamList.get(0);
// By default, new streams are from the AI agent, so play directly
ZegoExpressEngine.getEngine().startPlayingStream(stream.streamID);
}
}
});
}
Client request to play the AI agent stream
// Set up the event handler
void setEventHandler() {
ZegoExpressEngine.getEngine().setEventHandler(new IZegoEventHandler() {
@Override
// When other users in the room start/stop publishing streams, you can receive notifications about the corresponding user's audio/video stream changes here
public void onRoomStreamUpdate(String roomID, ZegoUpdateType updateType, ArrayList<ZegoStream> streamList, JSONObject extendedData) {
super.onRoomStreamUpdate(roomID, updateType, streamList, extendedData);
// When updateType is ZegoUpdateType.ADD, it means there is a new audio/video stream, at this time we can call the startPlayingStream interface to pull this audio/video stream
// !mark(1:8)
if (updateType == ZegoUpdateType.ADD) {
ZegoStream stream = streamList.get(0);
// By default, new streams are from the AI agent, so play directly
ZegoExpressEngine.getEngine().startPlayingStream(stream.streamID);
}
}
});
}
Client request to play the AI agent stream
// Listen for room stream information update status, and play the AI agent stream
- (void)onRoomStreamUpdate:(ZegoUpdateType)updateType
streamList:(NSArray<ZegoStream *> *)streamList
extendedData:(nullable NSDictionary *)extendedData
roomID:(NSString *)roomID{
if (updateType == ZegoUpdateTypeAdd) {
for (int i=0; i<streamList.count; i++) {
ZegoStream* item = [streamList objectAtIndex:i];
// !mark
[self startPlayStream:item.streamID];
}
} else if(updateType == ZegoUpdateTypeDelete) {
for (int i=0; i<streamList.count; i++) {
ZegoStream* item = [streamList objectAtIndex:i];
[[ZegoExpressEngine sharedEngine] stopPlayingStream:item.streamID];
}
}
}
Client request to play the AI agent stream
// Listen for room stream information update status, and play the AI agent stream
- (void)onRoomStreamUpdate:(ZegoUpdateType)updateType
streamList:(NSArray<ZegoStream *> *)streamList
extendedData:(nullable NSDictionary *)extendedData
roomID:(NSString *)roomID{
if (updateType == ZegoUpdateTypeAdd) {
for (int i=0; i<streamList.count; i++) {
ZegoStream* item = [streamList objectAtIndex:i];
// !mark
[self startPlayStream:item.streamID];
}
} else if(updateType == ZegoUpdateTypeDelete) {
for (int i=0; i<streamList.count; i++) {
ZegoStream* item = [streamList objectAtIndex:i];
[[ZegoExpressEngine sharedEngine] stopPlayingStream:item.streamID];
}
}
}
Client request to play the AI agent stream
ZegoExpressEngine.onRoomStreamUpdate = _onRoomStreamUpdate;
void _onRoomStreamUpdate(
String roomID,
ZegoUpdateType updateType,
List<ZegoStream> streamList,
Map<String, dynamic> extendedData,
) {
if (updateType == ZegoUpdateType.Add) {
for (var stream in streamList) {
ZegoExpressEngine.instance.startPlayingStream(stream.streamID);
}
} else if (updateType == ZegoUpdateType.Delete) {
for (var stream in streamList) {
ZegoExpressEngine.instance.stopPlayingStream(stream.streamID);
}
}
}
Client request to play the AI agent stream
ZegoExpressEngine.onRoomStreamUpdate = _onRoomStreamUpdate;
void _onRoomStreamUpdate(
String roomID,
ZegoUpdateType updateType,
List<ZegoStream> streamList,
Map<String, dynamic> extendedData,
) {
if (updateType == ZegoUpdateType.Add) {
for (var stream in streamList) {
ZegoExpressEngine.instance.startPlayingStream(stream.streamID);
}
} else if (updateType == ZegoUpdateType.Delete) {
for (var stream in streamList) {
ZegoExpressEngine.instance.stopPlayingStream(stream.streamID);
}
}
}
Client request to play the AI agent stream
// Listen to remote stream update events
function setupEvent() {
zg.on("roomStreamUpdate",
async (roomID, updateType, streamList) => {
if (updateType === "ADD" && streamList.length > 0) {
try {
for (const stream of streamList) {
// Play the AI agent stream
// !mark
const mediaStream = await zg.startPlayingStream(stream.streamID);
if (!mediaStream) return;
const remoteView = await zg.createRemoteStreamView(mediaStream);
if (remoteView) {
// A container with the id 'remoteSteamView' is required on the page to receive the AI agent stream [Reference Documentation]（https://www.zegocloud.com/article/api?doc=Express_Video_SDK_API~javascript_web~class~ZegoStreamView）
remoteView.play("remoteSteamView", {
enableAutoplayDialog: false,
});
}
}
} catch (error) {
console.error("Failed to pull stream:", error);
}
}
}
);
}
Client request to play the AI agent stream
// Listen to remote stream update events
function setupEvent() {
zg.on("roomStreamUpdate",
async (roomID, updateType, streamList) => {
if (updateType === "ADD" && streamList.length > 0) {
try {
for (const stream of streamList) {
// Play the AI agent stream
// !mark
const mediaStream = await zg.startPlayingStream(stream.streamID);
if (!mediaStream) return;
const remoteView = await zg.createRemoteStreamView(mediaStream);
if (remoteView) {
// A container with the id 'remoteSteamView' is required on the page to receive the AI agent stream [Reference Documentation]（https://www.zegocloud.com/article/api?doc=Express_Video_SDK_API~javascript_web~class~ZegoStreamView）
remoteView.play("remoteSteamView", {
enableAutoplayDialog: false,
});
}
}
} catch (error) {
console.error("Failed to pull stream:", error);
}
}
}
);
}
Client request to play the AI agent stream
// Listen to remote stream update events
function setupEvent() {
zg.on("roomStreamUpdate",
async (roomID, updateType, streamList) => {
if (updateType === "ADD" && streamList.length > 0) {
try {
for (const stream of streamList) {
// Set zego-player component attributes
const zegoPlayerAttr = {
componentID: `zego-${stream.streamID}`,
playerId: stream.streamID
};
// Update and render component list
this.setData({
zegoPlayerList: this.data.zegoPlayerList
})
const zegoPlayer = wxContext.selectComponent(`#${zegoPlayerAttr.componentID}`);
if (zegoPlayer) {
// !mark
await zg.startPlayingStream(zegoPlayer, stream.streamID);
}
}
} catch (error) {
console.error("Failed to pull stream:", error);
}
}
}
);
}
Client request to play the AI agent stream
// Listen to remote stream update events
function setupEvent() {
zg.on("roomStreamUpdate",
async (roomID, updateType, streamList) => {
if (updateType === "ADD" && streamList.length > 0) {
try {
for (const stream of streamList) {
// Set zego-player component attributes
const zegoPlayerAttr = {
componentID: `zego-${stream.streamID}`,
playerId: stream.streamID
};
// Update and render component list
this.setData({
zegoPlayerList: this.data.zegoPlayerList
})
const zegoPlayer = wxContext.selectComponent(`#${zegoPlayerAttr.componentID}`);
if (zegoPlayer) {
// !mark
await zg.startPlayingStream(zegoPlayer, stream.streamID);
}
}
} catch (error) {
console.error("Failed to pull stream:", error);
}
}
}
);
}
Congratulations🎉! After completing this step, you can ask the AI agent any question by voice, and the AI agent will answer your questions by voice!
Delete the agent instance and the user exits the room
The client calls the logout interface to exit the room and stops publishing and playing streams. At the same time, it notifies your server to end the call. After receiving the end call notification, your server will delete the AI agent instance, and the AI agent instance will automatically exit the room and stop publishing and playing streams. This completes a full interaction.
// Notify your server to end the call
private void stop() {
RequestBody body = RequestBody.create("", MediaType.parse("application/json; charset=utf-8"));
// !mark
Request request = new Request.Builder().url(YOUR_SERVER_URL + "/api/stop").post(body).build();
new OkHttpClient.Builder().build().newCall(request).enqueue(new Callback() {
@Override
public void onFailure(@NonNull Call call, @NonNull IOException e) {
}
@Override
public void onResponse(@NonNull Call call, @NonNull Response response) throws IOException {
if (response.isSuccessful()) {
// Exit room
// !mark
ZegoExpressEngine.getEngine().logoutRoom();
}
}
});
}
// Notify your server to end the call
private void stop() {
RequestBody body = RequestBody.create("", MediaType.parse("application/json; charset=utf-8"));
// !mark
Request request = new Request.Builder().url(YOUR_SERVER_URL + "/api/stop").post(body).build();
new OkHttpClient.Builder().build().newCall(request).enqueue(new Callback() {
@Override
public void onFailure(@NonNull Call call, @NonNull IOException e) {
}
@Override
public void onResponse(@NonNull Call call, @NonNull Response response) throws IOException {
if (response.isSuccessful()) {
// Exit room
// !mark
ZegoExpressEngine.getEngine().logoutRoom();
}
}
});
}
/**
* Notify your server to end the call
*
* @param completion Completion callback, returns operation result
* @discussion This method sends a request to the server to end the call, used to release the AI agent instance
*/
- (void)doStopCallWithCompletion:(void (^)(NSInteger code, NSString *message, NSDictionary *data))completion {
// Build request URL
// !mark
NSString *url = [NSString stringWithFormat:@"%@/api/stop", self.currentBaseURL];
NSURL *requestURL = [NSURL URLWithString:url];
// Create request
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:requestURL];
request.HTTPMethod = @"POST";
// Set request headers
[request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
// Create request parameters
NSMutableDictionary *params = [NSMutableDictionary dictionary];
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:params options:0 error:nil];
request.HTTPBody = jsonData;
// Create session
NSURLSession *session = [NSURLSession sharedSession];
// Send request
NSURLSessionDataTask *task = [session dataTaskWithRequest:request
completionHandler:^(NSData * _Nullable data,
NSURLResponse * _Nullable response,
NSError * _Nullable error) {
dispatch_async(dispatch_get_main_queue(), ^{
if (error) {
if (completion) {
completion(-1, @"Network request failed", nil);
}
return;
}
NSHTTPURLResponse *httpUrlResponse = (NSHTTPURLResponse *)response;
if (httpUrlResponse.statusCode != 200) {
if (completion) {
completion(httpUrlResponse.statusCode,
[NSString stringWithFormat:@"Server error: %ld", (long)httpUrlResponse.statusCode],
nil);
}
return;
}
NSError *jsonError;
NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data options:0 error:&jsonError];
if (jsonError) {
if (completion) {
completion(-2, @"Failed to parse response data", nil);
}
return;
}
// Parse response data
NSInteger code = [dict[@"code"] integerValue];
NSString *message = dict[@"message"];
NSDictionary *responseData = dict[@"data"];
if (completion) {
completion(code, message, responseData);
}
// Exit room
// !mark
[[ZegoExpressEngine sharedEngine] logoutRoom];
});
}];
[task resume];
}
/**
* Notify your server to end the call
*
* @param completion Completion callback, returns operation result
* @discussion This method sends a request to the server to end the call, used to release the AI agent instance
*/
- (void)doStopCallWithCompletion:(void (^)(NSInteger code, NSString *message, NSDictionary *data))completion {
// Build request URL
// !mark
NSString *url = [NSString stringWithFormat:@"%@/api/stop", self.currentBaseURL];
NSURL *requestURL = [NSURL URLWithString:url];
// Create request
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:requestURL];
request.HTTPMethod = @"POST";
// Set request headers
[request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
// Create request parameters
NSMutableDictionary *params = [NSMutableDictionary dictionary];
NSData *jsonData = [NSJSONSerialization dataWithJSONObject:params options:0 error:nil];
request.HTTPBody = jsonData;
// Create session
NSURLSession *session = [NSURLSession sharedSession];
// Send request
NSURLSessionDataTask *task = [session dataTaskWithRequest:request
completionHandler:^(NSData * _Nullable data,
NSURLResponse * _Nullable response,
NSError * _Nullable error) {
dispatch_async(dispatch_get_main_queue(), ^{
if (error) {
if (completion) {
completion(-1, @"Network request failed", nil);
}
return;
}
NSHTTPURLResponse *httpUrlResponse = (NSHTTPURLResponse *)response;
if (httpUrlResponse.statusCode != 200) {
if (completion) {
completion(httpUrlResponse.statusCode,
[NSString stringWithFormat:@"Server error: %ld", (long)httpUrlResponse.statusCode],
nil);
}
return;
}
NSError *jsonError;
NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data options:0 error:&jsonError];
if (jsonError) {
if (completion) {
completion(-2, @"Failed to parse response data", nil);
}
return;
}
// Parse response data
NSInteger code = [dict[@"code"] integerValue];
NSString *message = dict[@"message"];
NSDictionary *responseData = dict[@"data"];
if (completion) {
completion(code, message, responseData);
}
// Exit room
// !mark
[[ZegoExpressEngine sharedEngine] logoutRoom];
});
}];
[task resume];
}
// Notify your server to end the call
Future<Map<String, dynamic>> stopCall() async {
try {
final response = await http.post(
Uri.parse('$_currentBaseUrl/api/stop'),
headers: {'Content-Type': 'application/json'},
);
if (response.statusCode == 200) {
final json = jsonDecode(response.body);
return json;
}
return {'code': -1, 'message': 'Request failed'};
} catch (e) {
return {'code': -1, 'message': e.toString()};
}
}
/// Stop the conversation with the AI agent
Future<bool> stop() async {
stopCall();
final String _roomId = 'room_id_1';
final engine = ZegoExpressEngine.instance;
/// Stop publishing stream
await engine.stopPublishingStream();
/// Log out of the room
await engine.logoutRoom(_roomId);
return true;
}
// Notify your server to end the call
Future<Map<String, dynamic>> stopCall() async {
try {
final response = await http.post(
Uri.parse('$_currentBaseUrl/api/stop'),
headers: {'Content-Type': 'application/json'},
);
if (response.statusCode == 200) {
final json = jsonDecode(response.body);
return json;
}
return {'code': -1, 'message': 'Request failed'};
} catch (e) {
return {'code': -1, 'message': e.toString()};
}
}
/// Stop the conversation with the AI agent
Future<bool> stop() async {
stopCall();
final String _roomId = 'room_id_1';
final engine = ZegoExpressEngine.instance;
/// Stop publishing stream
await engine.stopPublishingStream();
/// Log out of the room
await engine.logoutRoom(_roomId);
return true;
}
// Exit room
async function stopCall() {
try {
// !mark
const response = await fetch(`${YOUR_SERVER_URL}/api/stop`, { // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
});
const data = await response.json();
console.log('End call result:', data);
return data;
} catch (error) {
console.error('Failed to end call:', error);
throw error;
}
}
stopCall();
zg.destroyLocalStream(localStream);
// !mark
zg.logoutRoom();
// Exit room
async function stopCall() {
try {
// !mark
const response = await fetch(`${YOUR_SERVER_URL}/api/stop`, { // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
});
const data = await response.json();
console.log('End call result:', data);
return data;
} catch (error) {
console.error('Failed to end call:', error);
throw error;
}
}
stopCall();
zg.destroyLocalStream(localStream);
// !mark
zg.logoutRoom();
// Exit room
async function stopCall() {
try {
// !mark
const response = await fetch(`${YOUR_SERVER_URL}/api/stop`, { // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
});
const data = await response.json();
console.log('End call result:', data);
return data;
} catch (error) {
console.error('Failed to end call:', error);
throw error;
}
}
stopCall();
zg.stopPublishingStream();
if (zegoPlayerList.length) {
for (const player of zegoPlayerList) {
zg.stopPlayingStream(player.playerId);
}
}
// !mark
zg.logoutRoom();
// Exit room
async function stopCall() {
try {
// !mark
const response = await fetch(`${YOUR_SERVER_URL}/api/stop`, { // YOUR_SERVER_URL is the address of your Your Server
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
});
const data = await response.json();
console.log('End call result:', data);
return data;
} catch (error) {
console.error('Failed to end call:', error);
throw error;
}
}
stopCall();
zg.stopPublishingStream();
if (zegoPlayerList.length) {
for (const player of zegoPlayerList) {
zg.stopPlayingStream(player.playerId);
}
}
// !mark
zg.logoutRoom();
This is the complete core process for you to achieve real-time voice interaction with an AI agent.
Best Practices for ZEGO Express SDK Configuration
To achieve the best audio call experience, it is recommended to configure the ZEGO Express SDK according to the following best practices. These configurations can significantly improve the quality of AI agent voice interactions.
Settings Before Join Room:
Enable traditional audio 3A processing (Acoustic Echo Cancellation AEC, Automatic Gain Control AGC, and Noise Suppression ANS)
Set the room usage scenario to High Quality Chatroom, as the SDK will adopt different optimization strategies for different scenarios
Set the audio device mode to default mode
Enable AI echo cancellation to improve echo cancellation effect (this feature requires contacting ZEGO technical support to obtain the corresponding version of ZEGOExpress SDK)
Configure volume ducking to avoid sound conflicts
Enable adaptive playback volume to enhance user experience
Enable AI noise reduction and set appropriate noise suppression level
ZegoEngineConfig config = new ZegoEngineConfig();
HashMap<String, String> advanceConfig = new HashMap<String, String>();
// Configure volume ducking to avoid sound conflicts
advanceConfig.put("set_audio_volume_ducking_mode", "1");
// Enable adaptive playback volume
advanceConfig.put("enable_rnd_volume_adaptive", "true");
config.advancedConfig = advanceConfig;
ZegoExpressEngine.setEngineConfig(config);
// Set room usage scenario to High Quality Chatroom
ZegoExpressEngine.getEngine().setRoomScenario(ZegoScenario.HIGH_QUALITY_CHATROOM);
// Set audio device mode to default mode
ZegoExpressEngine.getEngine().setAudioDeviceMode(ZegoAudioDeviceMode.GENERAL);
// Enable traditional audio 3A processing
ZegoExpressEngine.getEngine().enableAEC(true);
ZegoExpressEngine.getEngine().enableAGC(true);
ZegoExpressEngine.getEngine().enableANS(true);
// Enable AI echo cancellation
ZegoExpressEngine.getEngine().setAECMode(ZegoAECMode.AI_BALANCED);
// Enable AI noise reduction with moderate noise suppression
ZegoExpressEngine.getEngine().setANSMode(ZegoANSMode.MEDIUM);
ZegoEngineConfig config = new ZegoEngineConfig();
HashMap<String, String> advanceConfig = new HashMap<String, String>();
// Configure volume ducking to avoid sound conflicts
advanceConfig.put("set_audio_volume_ducking_mode", "1");
// Enable adaptive playback volume
advanceConfig.put("enable_rnd_volume_adaptive", "true");
config.advancedConfig = advanceConfig;
ZegoExpressEngine.setEngineConfig(config);
// Set room usage scenario to High Quality Chatroom
ZegoExpressEngine.getEngine().setRoomScenario(ZegoScenario.HIGH_QUALITY_CHATROOM);
// Set audio device mode to default mode
ZegoExpressEngine.getEngine().setAudioDeviceMode(ZegoAudioDeviceMode.GENERAL);
// Enable traditional audio 3A processing
ZegoExpressEngine.getEngine().enableAEC(true);
ZegoExpressEngine.getEngine().enableAGC(true);
ZegoExpressEngine.getEngine().enableANS(true);
// Enable AI echo cancellation
ZegoExpressEngine.getEngine().setAECMode(ZegoAECMode.AI_BALANCED);
// Enable AI noise reduction with moderate noise suppression
ZegoExpressEngine.getEngine().setANSMode(ZegoANSMode.MEDIUM);
ZegoEngineProfile* profile = [[ZegoEngineProfile alloc]init];
profile.appID = kZegoAppId;
profile.scenario = ZegoScenarioHighQualityChatroom; // High Quality Chatroom scenario, setting this scenario can avoid requesting camera permissions, integrators should set specific values according to their business scenarios
ZegoEngineConfig* engineConfig = [[ZegoEngineConfig alloc] init];
engineConfig.advancedConfig = @{
@"set_audio_volume_ducking_mode":@1,/** Configure volume ducking to avoid sound conflicts **/
@"enable_rnd_volume_adaptive":@"true",/** Enable adaptive playback volume **/
};
[ZegoExpressEngine setEngineConfig:engineConfig];
[ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
// Enable traditional audio 3A processing
[[ZegoExpressEngine sharedEngine] enableAGC:TRUE];
[[ZegoExpressEngine sharedEngine] enableAEC:TRUE];
[[ZegoExpressEngine sharedEngine] enableANS:TRUE];
// Enable AI echo cancellation
[[ZegoExpressEngine sharedEngine] setAECMode:ZegoAECModeAIBalanced];
// Enable AI noise reduction with moderate noise suppression
[[ZegoExpressEngine sharedEngine] setANSMode:ZegoANSModeMedium];
ZegoEngineProfile* profile = [[ZegoEngineProfile alloc]init];
profile.appID = kZegoAppId;
profile.scenario = ZegoScenarioHighQualityChatroom; // High Quality Chatroom scenario, setting this scenario can avoid requesting camera permissions, integrators should set specific values according to their business scenarios
ZegoEngineConfig* engineConfig = [[ZegoEngineConfig alloc] init];
engineConfig.advancedConfig = @{
@"set_audio_volume_ducking_mode":@1,/** Configure volume ducking to avoid sound conflicts **/
@"enable_rnd_volume_adaptive":@"true",/** Enable adaptive playback volume **/
};
[ZegoExpressEngine setEngineConfig:engineConfig];
[ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
// Enable traditional audio 3A processing
[[ZegoExpressEngine sharedEngine] enableAGC:TRUE];
[[ZegoExpressEngine sharedEngine] enableAEC:TRUE];
[[ZegoExpressEngine sharedEngine] enableANS:TRUE];
// Enable AI echo cancellation
[[ZegoExpressEngine sharedEngine] setAECMode:ZegoAECModeAIBalanced];
// Enable AI noise reduction with moderate noise suppression
[[ZegoExpressEngine sharedEngine] setANSMode:ZegoANSModeMedium];
Enable traditional audio 3A processing (Acoustic Echo Cancellation AEC, Automatic Gain Control AGC, and Noise Suppression ANS)
Set the room usage scenario to High Quality Chatroom, as the SDK will adopt different optimization strategies for different scenarios
Configure volume ducking to avoid sound conflicts
When pushing streams, configure the push parameters to automatically switch to available videoCodec
// Import necessary modules
import { ZegoExpressEngine } from "zego-express-engine-webrtc";
import { VoiceChanger } from "zego-express-engine-webrtc/voice-changer";
// Import the volume ducking plugin package. Use version 3.11.0-aiagent.187.187
// !mark(1:2)
import { AudioRendererManager } from "/AudioRendererManager/AudioRendererManager";
import { MediaElementRenderer } from "/AudioRendererManager/mediaElementRenderer";
// Load audio processing module, must be called before new ZegoExpressEngine
ZegoExpressEngine.use(VoiceChanger);
// Instantiate ZegoExpressEngine, set room usage scenario to High Quality Chatroom
// !mark
const zg = new ZegoExpressEngine(appid, server, { scenario: 7 })
// Traditional audio 3A processing is enabled by default in SDK
// Create local media stream
const localStream = await zg.createZegoStream();
// Configure volume ducking
zg.callExperimentalAPI({
method: "enableVolumeDucking",
params: {
localStream,
enable: true
}
});
// Pass the player into the volume ducking plugin package
let renderer;
zg.callExperimentalAPI({ method: "getAudioContext" }).then((ac) => {
const audioMgr = AudioRendererManager.create(ac);
// Get the audio element
const audio = document.querySelector("audio");
renderer = audioMgr.getMediaElementRenderer(audio);
});
// Register volume ducking update callback
zg.callExperimentalAPI({ method: "onVolumeDuckingUpdate", params: {} });
zg.on("recvExperimentalAPI", (result) => {
const { method, content } = result;
if (method === "onVolumeDuckingUpdate") {
console.warn("onVolumeDuckingUpdate", content.gain);
// Set the agent stream volume to content.gain returned by the callback. Assume remoteView is the remoteView you created in the step of pulling the agent stream.
remoteView.setVolume(content.gain * 100); // volume range [0,100]
// Set the player volume to content.gain returned by the callback
renderer.setVolume(content.gain * 100); // volume range [0,100]
}
});
// Push local media stream, need to set automatic switching to available videoCodec
await zg.startPublishingStream(userStreamId, localStream, {
// !mark
enableAutoSwitchVideoCodec: true,
});
// Check system requirements
async function checkSystemRequirements() {
// Check WebRTC support
const rtcSupport = await zg.checkSystemRequirements("webRTC");
if (!rtcSupport.result) {
console.error("Browser does not support WebRTC");
return false;
}
// Check microphone permission
const micSupport = await zg.checkSystemRequirements("microphone");
if (!micSupport.result) {
console.error("Microphone permission not granted");
return false;
}
return true;
}
// Import necessary modules
import { ZegoExpressEngine } from "zego-express-engine-webrtc";
import { VoiceChanger } from "zego-express-engine-webrtc/voice-changer";
// Import the volume ducking plugin package. Use version 3.11.0-aiagent.187.187
// !mark(1:2)
import { AudioRendererManager } from "/AudioRendererManager/AudioRendererManager";
import { MediaElementRenderer } from "/AudioRendererManager/mediaElementRenderer";
// Load audio processing module, must be called before new ZegoExpressEngine
ZegoExpressEngine.use(VoiceChanger);
// Instantiate ZegoExpressEngine, set room usage scenario to High Quality Chatroom
// !mark
const zg = new ZegoExpressEngine(appid, server, { scenario: 7 })
// Traditional audio 3A processing is enabled by default in SDK
// Create local media stream
const localStream = await zg.createZegoStream();
// Configure volume ducking
zg.callExperimentalAPI({
method: "enableVolumeDucking",
params: {
localStream,
enable: true
}
});
// Pass the player into the volume ducking plugin package
let renderer;
zg.callExperimentalAPI({ method: "getAudioContext" }).then((ac) => {
const audioMgr = AudioRendererManager.create(ac);
// Get the audio element
const audio = document.querySelector("audio");
renderer = audioMgr.getMediaElementRenderer(audio);
});
// Register volume ducking update callback
zg.callExperimentalAPI({ method: "onVolumeDuckingUpdate", params: {} });
zg.on("recvExperimentalAPI", (result) => {
const { method, content } = result;
if (method === "onVolumeDuckingUpdate") {
console.warn("onVolumeDuckingUpdate", content.gain);
// Set the agent stream volume to content.gain returned by the callback. Assume remoteView is the remoteView you created in the step of pulling the agent stream.
remoteView.setVolume(content.gain * 100); // volume range [0,100]
// Set the player volume to content.gain returned by the callback
renderer.setVolume(content.gain * 100); // volume range [0,100]
}
});
// Push local media stream, need to set automatic switching to available videoCodec
await zg.startPublishingStream(userStreamId, localStream, {
// !mark
enableAutoSwitchVideoCodec: true,
});
// Check system requirements
async function checkSystemRequirements() {
// Check WebRTC support
const rtcSupport = await zg.checkSystemRequirements("webRTC");
if (!rtcSupport.result) {
console.error("Browser does not support WebRTC");
return false;
}
// Check microphone permission
const micSupport = await zg.checkSystemRequirements("microphone");
if (!micSupport.result) {
console.error("Microphone permission not granted");
return false;
}
return true;
}
When pushing streams, set the push mode to RTC
// Import necessary modules
import { ZegoExpressEngine } from "zego-express-engine-webrtc";
// Instantiate ZegoExpressEngine
const zg = new ZegoExpressEngine(appID, server);
// Create local audio stream
const zegoPusher = this.selectComponent("#zegoPusher");
// Push local stream
// !mark
await zegoPusher.startPush(zg, userStreamId, {}, {enableCamera: false, mode: 'RTC'});
// Check system requirements
const checkSystemRequirements = async () => {
const rtc_sup = await zg.checkSystemRequirements();
if (res.code === 10002) {
// Microphone permission not enabled
return false;
}
if (res.code === 10001) {
// WeChat version too low to use related components
return false;
}
return true;
}
// Import necessary modules
import { ZegoExpressEngine } from "zego-express-engine-webrtc";
// Instantiate ZegoExpressEngine
const zg = new ZegoExpressEngine(appID, server);
// Create local audio stream
const zegoPusher = this.selectComponent("#zegoPusher");
// Push local stream
// !mark
await zegoPusher.startPush(zg, userStreamId, {}, {enableCamera: false, mode: 'RTC'});
// Check system requirements
const checkSystemRequirements = async () => {
const rtc_sup = await zg.checkSystemRequirements();
if (res.code === 10002) {
// Microphone permission not enabled
return false;
}
if (res.code === 10001) {
// WeChat version too low to use related components
return false;
}
return true;
}
Additional Optimization Recommendations
Browser Compatibility: Recommended to use the latest versions of modern browsers such as Chrome, Firefox, Safari
Network Environment: Ensure stable network connection, recommend using wired network or Wi-Fi with good signal
Audio Equipment: Use high-quality microphones and speakers
Page Optimization: Avoid running too many JavaScript tasks on the same page, which may affect audio processing performance
HTTPS Environment: Use HTTPS protocol in production environment to ensure microphone permission access
:::
Listen for Exception Callback
Note
Due to the large number of parameters for LLM and TTS, it is easy to cause various abnormal problems such as the AI agent not answering or not speaking during the test process due to parameter configuration errors. We strongly recommend that you listen for exception callbacks during the test process and quickly troubleshoot problems based on the callback information.
Receive Callback
Click to view the guide for listening to exception callbacks. The event with Event as Exception can be quickly located through Data.Code and Data.Message.
