Configuring LLM

Depending on your use case, you can plug in any third-party LLM—whether it's Volcano Ark, MiniMax, Qwen, Stepfun, DeepSeek, or your own in-house model. This guide walks you through configuring for the above kinds of LLMs and highlights key considerations.

LLM Parameter Description

When using third-party LLM services or custom LLM services, you need to configure LLM parameters.

Parameter	Type	Required	Description
Url	String	Yes	LLM callback address, which must be compatible with the OpenAI protocol.
ApiKey	String	No	Authentication credentials for accessing various models and related services provided by LLM.
Model	String	Yes	The model to call. Different LLM service providers support different configurations, please refer to the corresponding documentation.
SystemPrompt	String	No	System prompt. Can include role settings, prompts, and response examples.
Temperature	Float	No	Higher values will make the output more random, while lower values will make the output more focused and deterministic.
TopP	Float	No	Sampling method, smaller values result in stronger determinism; larger values result in more randomness.
Params	Object	No	Other LLM parameters, such as maximum Token number limit, etc. Different LLM providers support different configurations, please refer to the corresponding documentation and fill in as needed. Note Parameter names should match those of each vendor's LLM.
AddAgentInfo	Bool	No	If this value is true, when the AI Agent backend sends requests to custom LLM services, the request parameters will include agent information `agent_info`. This value defaults to false. When using custom LLM, additional business logic can be implemented based on this parameter content.

Using Third-party LLMs

Note

Please contact ZEGOCLOUD Technical Support first to activate third-party LLM services and obtain the access Url and API Key.

Third-party LLMs must be compatible with the OpenAI protocol.

You can set LLM parameters when registering an AI agent (RegisterAgent) or creating an AI agent instance (CreateAgentInstance).

Here are configuration samples for common LLM vendors:

Use Custom LLM

The ZEGOCLOUD AI Agent server uses the OpenAI API protocol to call LLM services. Therefore, you can also use any custom LLM compatible with the OpenAI protocol. The custom LLM can even call multiple sub-LLM models or perform RAG search and web search before integrating and outputting results at the underlying implementation level.

Implement Custom LLM

Create an interface that conforms to the OpenAI API protocol.

Register Agent and Use Custom LLM

When registering the agent (RegisterAgent), set the custom LLM URL, and require the LLM to answer the user's question based on the knowledge base content in the SystemPrompt.

Register Agent Call Example

// Please replace the LLM and TTS authentication parameters such as ApiKey, appid, token, etc. with your actual authentication parameters.
async registerAgent(agentId: string, agentName: string) {
    // Request interface: https://aigc-aiagent-api.zegotech.cn?Action=RegisterAgent
    const action = 'RegisterAgent';
    // !mark(4:9)
    const body = {
        AgentId: agentId,
        Name: agentName,
        LLM: {
            Url: "https://your-custom-llm-service/chat/completions",
            ApiKey: "your_api_key",
            Model: "your_model",
            SystemPrompt: "Please answer the user's question in a friendly manner based on the knowledge base content provided by the user. If the user's question is not in the knowledge base, please politely tell the user that we do not have related knowledge base content."
        },
        TTS: {
            Vendor: "ByteDance",
            Params: {
                "app": {
                    "appid": "zego_test",
                    "token": "zego_test",
                    "cluster": "volcano_tts"
                },
                "audio": {
                    "voice_type": "zh_female_wanwanxiaohe_moon_bigtts"
                }
            }
        }
    };
    // The sendRequest method encapsulates the request URL and public parameters. For details, please refer to: https://doc-zh.zego.im/aiagent-server/api-reference/accessing-server-apis
    return this.sendRequest<any>(action, body);
}

// Please replace the LLM and TTS authentication parameters such as ApiKey, appid, token, etc. with your actual authentication parameters.
async registerAgent(agentId: string, agentName: string) {
    // Request interface: https://aigc-aiagent-api.zegotech.cn?Action=RegisterAgent
    const action = 'RegisterAgent';
    // !mark(4:9)
    const body = {
        AgentId: agentId,
        Name: agentName,
        LLM: {
            Url: "https://your-custom-llm-service/chat/completions",
            ApiKey: "your_api_key",
            Model: "your_model",
            SystemPrompt: "Please answer the user's question in a friendly manner based on the knowledge base content provided by the user. If the user's question is not in the knowledge base, please politely tell the user that we do not have related knowledge base content."
        },
        TTS: {
            Vendor: "ByteDance",
            Params: {
                "app": {
                    "appid": "zego_test",
                    "token": "zego_test",
                    "cluster": "volcano_tts"
                },
                "audio": {
                    "voice_type": "zh_female_wanwanxiaohe_moon_bigtts"
                }
            }
        }
    };
    // The sendRequest method encapsulates the request URL and public parameters. For details, please refer to: https://doc-zh.zego.im/aiagent-server/api-reference/accessing-server-apis
    return this.sendRequest<any>(action, body);
}

You can now chat with your custom LLM.

Best Practices

Detailed usage cases please refer to Use AI Agent with RAG.