logo
On this page

Display Subtitles


This article introduces how to display subtitles during a voice call between a user and an AI agent. As follows:

  • User's speech: Stream the user's spoken content as it is being recognized by ASR in real time.
  • AI agent's speech: Stream the AI agent's output content as it is being generated by LLM in real time.
image.png

Prerequisites

You should have already integrated the ZEGO Express SDK and the ZEGOCLOUD AI Agent, and implemented a basic voice-call feature following the Quick Start doc.

Quick Implementation

During voice conversations between users and AI agents, the ZEGOCLOUD AI Agent server sends ASR recognition text and LLM response text via custom messages in the RTC room to the client. By listening for these custom messages, the client can parse the status events and render the UI.

The processing flowchart for RTC room custom messages is as follows:

Cmd=3

Cmd=4

Start

Implement onRecvExperimentalAPI callback and initialize subtitle UI component

Parse RTC room custom messages

Process ASR text

Process LLM text

Sort and update user subtitles

Sort and update AI agent subtitles

Clear message cache after message ends

End

Listening to Custom Room Messages

By listening to the recvExperimentalAPI callback, the client can obtain room custom messages with method as onRecvRoomChannelMessage. Below is an example of the callback listener code:

Untitled
// WARNING!!!: The data received through custom room messages may be out of order, and sorting needs to be performed based on the SeqId field.
zg.on("recvExperimentalAPI", (result) => {
  const { method, content } = result;
  if (method === "onRecvRoomChannelMessage") {
    try {
      // Parse the message
      const recvMsg = JSON.parse(content.msgContent);
      const { Cmd, SeqId, Data, Round } = recvMsg;
    } catch (error) {
      console.error("Failed to parse the message:", error);
    }
  }
});
// Enable the experimental API for onRecvRoomChannelMessage
zg.callExperimentalAPI({ method: "onRecvRoomChannelMessage", params: {} });
123456789101112131415
Copied!

Room Custom Message Protocol

The fields of the room custom message are described as follows:

FieldTypeDescription
TimestampNumberTimestamp, at the second level
SeqIdNumberPacket sequence number, may be out of order. Please sort the messages according to the sequence number. In extreme cases, the Id may not be continuous.
RoundNumberDialogue round, increases each time the user starts speaking
CmdNumber
  • 3: ASR text.
  • 4: LLM text.
DataObjectSpecific content, different Cmds correspond to different Data

Data varies depending on the Cmd as follows:

FieldTypeDescription
TextStringASR text of user speech.
Each issuance is the full text, supporting text correction.
MessageIdStringMessage ID. It is unique for each round of ASR text message.
EndFlagBoolEnd flag, true indicates that the ASR text of this round has been processed.

Processing Logic

Determine the message type based on the Cmd field, and obtain the message content from the Data field.

Untitled
 // Handle user message
 function handleUserMessage(data, seqId, round) {
    if (data.EndFlag) {
      // User has finished speaking
    }
    const content = data.Text;
    if (content) {
      // Use the ASR text corresponding to the latest seqId as the latest speech recognition result and update the UI
    }
 }
12345678910
Copied!

The corresponding message processing flow is shown in the figure below:

image.png

Use the subtitle component

If you are working on a Vue project, you can download the subtitle component to your project and use it directly.

Precautions

  • Message Sorting Processing: The data received through custom room messages may be out of order, and sorting needs to be performed based on the SeqId field.
  • Streaming Text Processing:
  • Each ASR text sent is the full text. Messages with the same MessageId should completely replace the previous content.
  • Each LLM text sent is incremental. Messages with the same MessageId need to be cumulatively displayed after sorting.
  • Memory Management: Please clear the cache of completed messages in time, especially when users engage in long conversations.

Previous

Quick Start with Digital Human