Controlling AI Agent Speech Emotion

2026-07-09

Feature Description

In AI real-time voice interaction scenarios, the default TTS voice is often flat and lacks emotional variation. By controlling parameters such as emotion and tone during TTS synthesis, you can make the AI's voice more expressive and immersive.

For example, in a customer service scenario, the AI can automatically switch to a warm, patient tone based on the user's questions; in a gaming companion scenario, the AI can express excitement, surprise, and other emotions as the conversation progresses. These capabilities are controlled through TTS synthesis parameters.

There are two main control approaches:

Specific emotion enums: Control voice emotion through predefined emotion tags (such as happy, sad, angry). This approach uses emotion lists defined by the TTS vendor, and developers select from them. For example, MiniMax's Speech series supports specifying emotions such as happy, sad, and angry for speech synthesis. You can experience it through the Voice Debugging Console.
Natural language descriptions: Control voice style through natural language instructions (such as "Speak in a gentle tone"). This approach is more flexible and can describe subtler emotional variations. For example, Volcengine Doubao TTS 2.0 allows entering instructions like "Speak in a gentle tone" to control the synthesis effect. You can experience it through Viewing voice instruction and tag documentation.

For more TTS vendors currently supported by ZEGO and their corresponding capabilities, refer to Appendix - Supported Vendors and Parameters.

Feature Flowchart

Using Doubao TTS 2.0's natural language description (context_texts tag), calling "Unidirectional Streaming TTS WebSocket" through ZEGO TTS-Vendor=ByteDanceV3 as an example.

Specific Usage Steps

Example

Using Doubao TTS 2.0's natural language description (context_texts tag), calling "Unidirectional Streaming TTS WebSocket" through ZEGO TTS-Vendor=ByteDanceV3 as an example, to achieve the effect of the AI automatically determining and broadcasting its reply with a specified tone + pitch.

Prerequisites

Enable the AI Agent service.
Confirm that the TTS model or timbre used supports the corresponding synthesis parameters, and that the ZEGO AI Agent service supports them (see the "Appendix - Supported Vendors and Parameters" section for details).

Implementation Steps

Register Agent / Create Agent Instance

Complete the following configuration through the Register Agent or Create Agent Instance API.

Configuration 1: Configure LLMMetaInfo Extraction Rules

Configure the AdvancedConfig.LLMMetaInfo parameter of the Create Agent Instance API to specify how to extract metadata from LLM text. For example:

"AdvancedConfig": {
  "LLMMetaInfo": {
    "BeginCharacters": "[[",
    "EndCharacters": "]]"
  }
}

"AdvancedConfig": {
  "LLMMetaInfo": {
    "BeginCharacters": "[[",
    "EndCharacters": "]]"
  }
}

No need to configure the FilterText.BeginCharacters and FilterText.EndCharacters parameters again when registering the agent or creating the agent instance, because the metadata and marker symbols will be removed from the LLM output text.

Configuration 2: Configure TTSParamPaths Mapping

Various TTS vendors can control the synthesized voice's emotional effect through certain parameters. TTSParamPaths is a mapping table passed through AdvancedConfig when creating an agent instance. This parameter controls how the metadata markers in the LLM output content are mapped to TTS emotion control parameters.

Taking Volcengine TTS (Unidirectional Streaming) as an example:

"AdvancedConfig": {
  "TTSParamPaths": {
    "context_texts": "req_params.additions.context_texts",
    "pitch": "req_params.additions.post_process.pitch"
  }
}

"AdvancedConfig": {
  "TTSParamPaths": {
    "context_texts": "req_params.additions.context_texts",
    "pitch": "req_params.additions.post_process.pitch"
  }
}

This means that when the LLM output is [[{"context_texts":"Speak in a gentle tone","pitch":"1"}]]Hello, the metadata in this output will be extracted:

The context_texts value Speak in a gentle tone from {"context_texts":"Speak in a gentle tone"}
The pitch value 1 from "pitch":"1"

And passed as values to the TTS vendor API's req_params.additions.context_texts and req_params.additions.post_process.pitch.

Description

The keys in TTSParamPaths can be customized based on business logic (for example, changing instruction to i), as long as the metadata keys in the LLM output are consistent with the keys defined in TTSParamPaths.

Configuration 3: Have the LLM Output Content in the Specified Format

In the LLM.SystemPrompt of the Register Agent or Create Agent Instance API, prompt the LLM to output tags containing native vendor parameters in its responses.

The following is a snippet example that can be added to SystemPrompt. Please adjust according to your actual business needs:

## Voice Emotion Control Rules
At the very beginning of each reply, output a JSON tag to control the voice emotion of the current reply, in the following format:
[[{"context_texts":["Enter the desired tone description here, e.g.: Speak in a gentle tone"],"pitch":Enter an integer from 1-5}]]

Description:
- context_texts: Use natural language to describe the voice style you want for this reply (e.g., "gentle tone", "excited tone").
- pitch: Pitch adjustment, an integer from 1-5; the higher the value, the higher the pitch. If not needed, this field can be omitted.
- The tag must be placed at the very beginning of the reply text, and the tag content will not be read aloud.
- Immediately after the tag, output the normal reply text to the user.

Example:
Input: User says "Hello"
Your output should be: [[{"context_texts":["Speak in a gentle tone"],"pitch":5}]]Hello there, nice to meet you!

## Voice Emotion Control Rules
At the very beginning of each reply, output a JSON tag to control the voice emotion of the current reply, in the following format:
[[{"context_texts":["Enter the desired tone description here, e.g.: Speak in a gentle tone"],"pitch":Enter an integer from 1-5}]]

Description:
- context_texts: Use natural language to describe the voice style you want for this reply (e.g., "gentle tone", "excited tone").
- pitch: Pitch adjustment, an integer from 1-5; the higher the value, the higher the pitch. If not needed, this field can be omitted.
- The tag must be placed at the very beginning of the reply text, and the tag content will not be read aloud.
- Immediately after the tag, output the normal reply text to the user.

Example:
Input: User says "Hello"
Your output should be: [[{"context_texts":["Speak in a gentle tone"],"pitch":5}]]Hello there, nice to meet you!

Trigger AI to Speak

Trigger Method 1: User Speaks

The user speaks in a real-time voice conversation. The voice is transmitted via RTC, transcribed by ASR, and input to the LLM. The LLM automatically generates a reply and outputs [[tag]] at the beginning of the sentence according to the configured SystemPrompt. The tag enters the ZEGO processing flow along with the LLM streaming output, without the need for additional API calls.

Trigger Method 2: Use sendAgentInstanceLLM to have the AI proactively speak

The LLM itself does not support proactive output. Developers need to proactively trigger the agent to speak based on certain rules to enhance the immersive experience in real-time interactions. For example, when the user has not spoken for 5 seconds, have the agent proactively start a conversation.

Use sendAgentInstanceLLM to proactively trigger the LLM to generate a reply. The LLM similarly outputs text with [[tag]] at the beginning of the reply according to the configured SystemPrompt, and the processing flow is the same as Trigger Method 1.

Example of calling the sendAgentInstanceLLM API:

{
    "AgentInstanceId": "1907755175297171456",
    "Text": "Proactively greet the user"
}

{
    "AgentInstanceId": "1907755175297171456",
    "Text": "Proactively greet the user"
}

After the call, the LLM will generate a reply based on the context and output [[tag]] at the beginning of the reply according to the configured SystemPrompt. The subsequent extraction, mapping, and synthesis flow is completely identical to Trigger Method 1.

LLM SSE Data Return Example:

data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":"[[{\"context_texts\":[\"Speak in a gentle tone\"],\"pitch\":5}]]"},"finish_reason":""}],"usage":{"prompt_tokens":83,"completion_tokens":1,"total_tokens":84}}
data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":"Hello"},"finish_reason":""}],"usage":{"prompt_tokens":83,"completion_tokens":2,"total_tokens":85}}
data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":" there"},"finish_reason":""}],"usage":{"prompt_tokens":83,"completion_tokens":3,"total_tokens":86}}
...
data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":"."},"finish_reason":"stop"}],"usage":{"prompt_tokens":83,"completion_tokens":150,"total_tokens":233}}
data: [DONE]

data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":"[[{\"context_texts\":[\"Speak in a gentle tone\"],\"pitch\":5}]]"},"finish_reason":""}],"usage":{"prompt_tokens":83,"completion_tokens":1,"total_tokens":84}}
data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":"Hello"},"finish_reason":""}],"usage":{"prompt_tokens":83,"completion_tokens":2,"total_tokens":85}}
data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":" there"},"finish_reason":""}],"usage":{"prompt_tokens":83,"completion_tokens":3,"total_tokens":86}}
...
data: {"id":"d7ae7c4a-1524-4fe5-9d58-e4d59b89d8f0","object":"chat.completion.chunk","created":1709899323,"model":"step-1-8k","choices":[{"index":0,"delta":{"role":"","content":"."},"finish_reason":"stop"}],"usage":{"prompt_tokens":83,"completion_tokens":150,"total_tokens":233}}
data: [DONE]

Trigger Method 3: Use sendAgentInstanceTTS to have the AI proactively speak (Optional)

In addition to automatic tag output through the LLM, you can also directly include tags at the very beginning of the text field when calling APIs such as sendAgentInstanceTTS. The tag content will not be read aloud:

{
  "text": "[[{\"context_texts\":[\"Speak in a gentle tone\"],\"pitch\":5}]]Hello"
}

{
  "text": "[[{\"context_texts\":[\"Speak in a gentle tone\"],\"pitch\":5}]]Hello"
}

The tag must be placed at the very beginning of the text (at the start of the sentence).

ZEGO Automatically Processes and Controls TTS Playback

Regardless of which trigger method is used above, the ZEGO AI Agent service will process according to established rules: ① Extract [[tag]] according to LLMMetaInfo and remove the markers → ② Map short parameter names to vendor request body paths according to TTSParamPaths → ③ Call the third-party TTS to synthesize with the parameters.

The end user hears emotionally expressive voice, for example, broadcasting "Hello" with a "gentle tone + pitch=5".

How to Adapt to Other TTS Vendors

This example uses Volcengine ByteDanceV3 as an example. If you use another vendor, you only need to adjust the following three places, and the rest of the flow (tag extraction, trigger methods, ZEGO automatic processing) remains completely unchanged:

The TTSParamPaths mapping path in Configuration 2 — Replace with the target vendor's parameter path. See the "TTS Parameter Path" column in the appendix table Supported Vendors and Parameters. Example: Aliyun CosyVoice's instruction → payload.parameters.instruction
The SystemPrompt prompt in Configuration 3 — Replace with the parameter names and value rules supported by the target vendor. Example: For MiniMax, change to output emotion enum; for CosyVoice, change to output instruction command
The text tag in Trigger Method 3 (if using direct TTS calls) — Simply replace the field names in the tag with the target vendor's parameter names

Special Notes

Customers are now responsible for managing the validity of the parameters they pass in. If the LLM generates or the sendAgentInstanceTTS call passes parameter values not supported by the vendor, the generated TTS may have no audio output.
The timbre/emotion list may change. Please refer to the latest list provided by the TTS vendor.

Legacy Approach (Deprecated, for reference by existing users only)

The following approach is an earlier implementation. It is recommended to use the TTSParamPaths-based native vendor parameter pass-through method described above.

1. Specify the Format of the Content in the LLM Text that Controls Emotion

Configure the AdvancedConfig.LLMMetaInfo parameter of the Create Agent Instance API to specify how to extract the metadata controlling emotion from the LLM text. For example:

"LLMMetaInfo" : {
    "BeginCharacters": "[[",
    "EndCharacters": "]]"
}

"LLMMetaInfo" : {
    "BeginCharacters": "[[",
    "EndCharacters": "]]"
}

2. Have the LLM Output Content According to the Specified Emotion Control Format

In the following example, the emotion values are only examples. The actual emotions supported by the TTS vendor can be included in the text output by the LLM. However, generally not all emotions are included (for example, a customer service application will not allow it to have a sad emotion).

The following are the LLM.SystemPrompt examples corresponding to the Register Agent and Create Agent Instance APIs when using MiniMax and Doubao TTS, for reference only. Please adjust according to actual needs:

3. Have the TTS Vendor Synthesize Emotionally Expressive Voice Based on Emotion Control Parameters

Now you can start a voice conversation with the created agent instance! When the content output by the LLM contains emotion control parameters, the AI Agent service will automatically call the TTS vendor interface based on these parameters, allowing it to interact with you with rich emotional voice expression.

Appendix

Supported Vendors and Parameters

ZEGO AI Agent currently supports the following TTS vendors' emotion control capabilities. Different vendors and models support different control methods: MiniMax and Volcengine Doubao TTS 1.0 support specifying predefined emotion enums through the emotion tag; Volcengine Doubao TTS 2.0 and CosyVoice support describing voice style through natural language instructions (such as instruction or context_texts).

The table below lists the pass-through parameters supported by each vendor. The "TTS Parameter Path" column contains the value (dot-notation path of the vendor request body) needed for the TTSParamPaths mapping table, while the short parameter names used in tags correspond to the key of TTSParamPaths.

Vendor (TTS-Vendor)	Applicable Model	TTS Parameter Path	Type	Supported Vendor Parameter Description	LLM Output	Direct TTS Call
Volcengine TTS (Unidirectional Streaming) `ByteDanceV3`	1.0 Series	`req_params.audio_params.emotion`	string	Specific emotion enum, such as `happy`, `sad`, `angry`, `fearful`, `surprised`, `neutral`, etc. View emotion parameter documentation / Doubao TTS Model	✅	✅
Volcengine TTS (Unidirectional Streaming) `ByteDanceV3`	1.0 Series	`req_params.audio_params.emotion_scale`	number	Emotion intensity 1-5 Note: Only supported by 1.0 series	✅	✅
Volcengine TTS (Unidirectional Streaming) `ByteDanceV3`	2.0 Series	`req_params.additions.context_texts`	string[]	Natural language instruction, such as entering instructions like "Speak in a gentle tone" or "Speak in an excited tone" to control voice style. View voice instruction and tag documentation / Multiple emotional timbres in Doubao TTS Model Note: Only supported by 2.0 series	✅	✅
Volcengine TTS (Unidirectional Streaming) `ByteDanceV3`	1.0 / 2.0	`req_params.additions.post_process.pitch`	int	Pitch adjustment	✅	✅
Volcengine TTS (Bidirectional Streaming) `ByteDanceFlowing`	2.0 Series	`req_params.additions.context_texts`	string[]	Natural language instruction View voice instruction and tag documentation / Multiple emotional timbres in Doubao TTS Model Note: Only supported by 2.0 series	❌	✅
Aliyun CosyVoice `CosyVoice`	CosyVoice	`payload.parameters.instruction`	string	Natural language description, such as entering instructions like "Speak in a gentle tone" or "Speak in a Sichuan dialect" to control dialect, emotion, role, etc. View instruction control documentation / CosyVoice Experience	✅	✅
MiniMax `MiniMax`	Speech Series	`voice_setting.emotion`	string	Specific emotion enum, such as `happy`, `sad`, `angry`, `fearful`, `surprised`, `neutral`, etc. View full emotion list / Voice Debugging Console	✅	❌

Description

Volcengine TTS bidirectional streaming does not support the LLM streaming output scenario: inserting tags after LLM sentence completion may cause sentence breaks. To use it in bidirectional streaming, please pass tags via direct TTS API calls (e.g., sendAgentInstanceTTS).