Quick Start Voice Call
Quick Start Digital Human Video Call
This document explains how to quickly call AI Agent related backend APIs to achieve video interaction with AI Agent.
Digital Human Introduction
With just a photo or image of a real person or anime character from the waist up, you can obtain a 1080P digital human with accurate lip-sync and realistic appearance. When used with the AI Agent product, you can quickly achieve video interaction chat with AI digital humans with an overall latency of about 1.5s, suitable for scenarios such as digital human 1V1 interactive video, digital human customer service, and digital human live streaming.
- More natural driving effects: supports subtle body movements, natural facial expressions without distortion, providing a more realistic and immersive interaction compared to voice calls;
- Multi-language accurate lip-sync: natural and accurate lip movements, especially optimized for Chinese and English;
- Ultra-low interaction latency: digital human driving latency < 200ms, combined with AI Agent interaction latency < 1.5s;
- Higher clarity: true 1080P effect, 20%+ improvement in clarity compared to traditional image-based digital humans.
Go to the download page to download and experience.
Quick Start Digital Human Live Broadcast
Differences from Digital Human Video Calls
Prerequisites
- You have created a project in the ZEGOCLOUD Console and obtained a valid AppID and ServerSecret (for server API signing and RTC Token04 generation). For details, see Console - Project Info.
- You have contacted ZEGOCLOUD Technical Support to enable the AI Agent related services and obtained LLM and TTS related configuration information. For details, see Console - AI Agent.
- You have obtained a valid
digital_human_id(for testing, you can use the public ID: 63c3aa64-1d80-4b04-a0be-1c65614eb7eb).
- You have contacted ZEGOCLOUD Technical Support to create a digital human.
During the test period (within 2 weeks after the AI Agent service is enabled), you can set the LLM and TTS authentication parameters to "zego_test" to use the related services. For details, see Register Agent > BODY Parameter Description.
You can also purchase LLM and TTS services supported by ZEGOCLOUD to obtain authentication information. Alternatively, you can contact ZEGOCLOUD sales to purchase TTS services directly.
Example Code
The following is the example code for the business backend that integrates the real-time interactive AI Agent API. You can refer to the example code to implement your own business logic.
Includes the basic capabilities of obtaining ZEGO Token, registering AI Agent, creating AI Agent instances, and deleting AI Agent instances.
The following is the client sample code. You can refer to the example code to implement your own business logic.
Includes the basic capabilities of logging in, publishing stream, playing stream, and exiting rooms.
Includes the basic capabilities of logging in, publishing stream, playing stream, and exiting rooms.
Includes the basic capabilities of logging in, publishing stream, playing stream, and exiting rooms.
Includes the basic capabilities of logging in, publishing stream, playing stream, and exiting rooms.
The following video demonstrates how to run the service backend and client (Web) example code and interact with the digital human agent.
Overall Business Process
- Service backend: run the business backend example code and deploy the business backend.
- Integrate the real-time interactive AI Agent API to manage AI Agents.
- Client: refer to the Android Quick Start, iOS Quick Start, or Web Quick Start document to run the client example code.
- Create and manage AI Agents through the business backend.
- Integrate ZEGO Express SDK for real-time communication.
- Integrate Digital Human SDK for digital human rendering.
After completing the above three steps, you can achieve real-time video interaction between the AI Agent and real users by joining the room.
Core Implementation
Register Agent
Register Agent is used to set the basic configuration of the AI Agent, including the agent name, LLM, TTS, ASR, and other related configurations. After registration, the agent can be used as a template to create multiple instances for real-time interaction with multiple real users.
Typically, the agent configuration is relatively fixed. Once the relevant parameters (persona) are set, they do not change frequently. Therefore, it is recommended to register the agent at the appropriate time according to your business flow. The agent will not be automatically destroyed or recycled after registration. Once an agent instance is created, you can interact with the agent via voice.
The following is an example of calling the Register Agent API:
// Please replace the LLM and TTS authentication parameters (ApiKey, appid, token, etc.) in the following example with your actual authentication parameters.
async registerAgent(agentId: string, agentName: string) {
// Request URL: https://aigc-aiagent-api.zegotech.cn?Action=RegisterAgent
const action = 'RegisterAgent';
const body = {
AgentId: agentId,
Name: agentName,
LLM: {
Url: "https://ark.cn-beijing.volces.com/api/v3/chat/completions",
ApiKey: "zego_test",
Model: "doubao-1-5-pro-32k-250115",
SystemPrompt: "You are an AI Agent. Please answer the user's questions."
},
TTS: {
Vendor: "ByteDance",
Params: {
"app": {
"appid": "zego_test",
"token": "zego_test",
"cluster": "volcano_tts"
},
"audio": {
"voice_type": "zh_female_wanwanxiaohe_moon_bigtts"
}
}
}
};
// The sendRequest method encapsulates the request URL and common parameters. For details, see: /aiagent-server/api-reference/accessing-server-apis
return this.sendRequest<any>(action, body);
} - Please ensure all LLM parameters are correctly filled in according to the LLM service provider's official documentation. Otherwise, you may not be able to see the agent's text responses or hear the agent's voice output.
- Please ensure all TTS parameters are correctly filled in according to the TTS service provider's official documentation. Otherwise, you may see the agent's text responses but not hear the agent's voice output.
- If the agent cannot output text or voice, please first check whether the LLM and TTS parameter configurations are completely correct, or refer to Get AI Agent Service Status - Monitor Server Exception Events to identify the specific issue.
Create Digital Human Agent Instance
You can use a registered agent as a template to create multiple digital human agent instances to join different rooms for real-time interaction with different users. After creating a digital human agent instance, the instance will automatically log in to the room and publish stream, while also playing the real user's stream.
After successfully creating a digital human agent instance, the digital human config needs to be returned to the client. The client initializes the Digital Human SDK based on the digital human config, and can then interact with the digital human in real time.
The following is an example of calling the Create Digital Human Agent Instance API:
async createDigitalHumanAgentInstance(agentId: string, userId: string, rtcInfo: RtcInfo, digitalHuman: DigitalHumanInfo, messages?: any[]) {
// Request URL: https://aigc-aiagent-api.zegotech.cn?Action=CreateDigitalHumanAgentInstance
// const rtcInfo = {
// RoomId: room_id,
// AgentStreamId: agent_stream_id,
// AgentUserId: agent_user_id,
// UserStreamId: user_stream_id,
// };
const action = 'CreateDigitalHumanAgentInstance';
const body = {
AgentId: agentId,
UserId: userId, // Real user ID interacting with this AI Agent instance
RTC: rtcInfo,
DigitalHuman: digitalHuman, // For testing, you can use the public ID: 63c3aa64-1d80-4b04-a0be-1c65614eb7eb
MessageHistory: {
SyncMode: 1, // Change to 0 to use history messages from ZIM
Messages: messages && messages.length > 0 ? messages : [],
WindowSize: 10
}
};
// The sendRequest method encapsulates the request URL and common parameters. For details, see: /aiagent-server/api-reference/accessing-server-apis
const result = await this.sendRequest<any>(action, body);
console.log("create agent instance result", result);
// The returned DigitalHumanConfig is the digital human config. The client initializes the Digital Human SDK based on the config, and can then interact with the digital human in real time.
return result.AgentInstanceId, result.DigitalHumanConfig;
} After completing this step, you can create digital human agent instances. Once the client is integrated, you can interact with the digital human agent instance via video.
Integrate Client SDK
Please refer to the following documents to complete client integration development:
Quick Start
Quick Start
Quick Start
Congratulations 🎉! After completing this step, you have successfully integrated the client SDK and can interact with the agent instance in real time via voice. You can ask the agent any question by voice, and the agent will respond!
Delete Agent Instance
After deleting the agent instance, the agent instance will automatically exit the room and stop publishing stream. After the real user stops publishing stream and exits the room on the client, a complete interaction session ends.
The following is an example of calling the Delete Agent Instance API:
async deleteAgentInstance(agentInstanceId: string) {
// Request URL: https://aigc-aiagent-api.zegotech.cn?Action=DeleteAgentInstance
const action = 'DeleteAgentInstance';
const body = {
AgentInstanceId: agentInstanceId
};
// The sendRequest method encapsulates the request URL and common parameters. For details, see: /aiagent-server/api-reference/accessing-server-apis
return this.sendRequest(action, body);
} The above is the complete core flow for achieving real-time voice interaction with the AI Agent.
Callback Monitoring
Click to view the callback monitoring guide. Monitor events where Event is Exception in the callbacks. You can quickly locate issues through Data.Code and Data.Message.
Click to view the exception error code list.