Realtime (WebSocket) models

Realtime models do not use /v1/chat/completions; they are called over a long-lived WebSocket connection. The gateway relays the Bailian Qwen-Omni-Realtime protocol verbatim in both directions — there is no protocol translation: you send Bailian events and receive Bailian events. This page covers connection, a runnable minimal example, and billing; the exact event and error schema is in the API reference.

What it does

• Live voice chat: speak and get streamed text and audio back as you go.
• Interruption: when the user speaks again the server emits input_audio_buffer.speech_started; the client stops playback and sends response.cancel to interrupt the current reply.
• Omni-modal input: streaming audio, images (frames sampled from video), and plain text.
• Text + audio output: the model produces both text and speech.

Supported models

A logical model can use this surface only if its catalog public_protocols includes dashscope_realtime. The realtime model currently available is qwen3.8-omni-flash-realtime (check the marketplace for what is actually listed). Check the model marketplace at https://console.modelgo.ai to confirm whether a model offers this protocol.

Connecting

Handshake failures are returned as HTTP status codes (e.g. an invalid key is 401 unauthorized, a model without the realtime protocol is 403 model_protocol_not_offered, insufficient balance is 402); see the full list in the API reference.

Minimal example

The examples below run server-side: connect with an mgk_ API key, configure the session, push 16 kHz PCM audio, commit, request a reply, and write the returned 24 kHz PCM to reply.pcm.

Prepare input: capture the microphone, or save 16 kHz, 16-bit little-endian, mono, headerless PCM as input.pcm. If you have a WAV, extract the raw PCM first:

import wave
from pathlib import Path

with wave.open("input.wav", "rb") as f:
    assert (f.getnchannels(), f.getframerate(), f.getsampwidth(), f.getcomptype()) == (1, 16000, 2, "NONE")
    Path("input.pcm").write_bytes(f.readframes(f.getnframes()))

Python (pip install websocket-client):

import base64
import json
import os
from pathlib import Path
import websocket

API_KEY = os.environ["MODELGO_API_KEY"]
MODEL = "qwen3.8-omni-flash-realtime"
URL = "{{ ws_endpoint }}/api-ws/v1/realtime?model=" + MODEL

audio = Path("input.pcm").read_bytes()  # 16 kHz / 16-bit LE / mono / headerless
if not audio or len(audio) % 2:
    raise ValueError("provide non-empty 16-bit mono PCM audio")

ws = websocket.create_connection(
    URL, header=["Authorization: Bearer " + API_KEY], timeout=30
)

def recv():
    ev = json.loads(ws.recv())
    if ev["type"] == "error":
        raise RuntimeError(ev["error"])
    return ev

try:
    while recv()["type"] != "session.created":
        pass
    ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "modalities": ["text", "audio"],
            "turn_detection": None,
            "audio": {
                "input": {"format": {"type": "pcm", "sample_rate": 16000}},
                "output": {"format": {"type": "pcm", "sample_rate": 24000}},
            },
        },
    }))
    while recv()["type"] != "session.updated":
        pass
    for i in range(0, len(audio), 3200):  # 100ms per frame
        ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": base64.b64encode(audio[i:i + 3200]).decode("ascii"),
        }))
    ws.send(json.dumps({"type": "input_audio_buffer.commit"}))
    while recv()["type"] != "input_audio_buffer.committed":
        pass
    ws.send(json.dumps({"type": "response.create"}))
    with open("reply.pcm", "wb") as out:  # 24 kHz / 16-bit LE / mono
        while True:
            ev = recv()
            if ev["type"] == "response.audio.delta":
                out.write(base64.b64decode(ev["delta"]))
            elif ev["type"] in ("response.audio_transcript.delta", "response.text.delta"):
                print(ev["delta"], end="", flush=True)
            elif ev["type"] == "response.done":
                print("\nusage:", ev["response"].get("usage"))
                break
finally:
    ws.close()

JavaScript / Node.js (npm install ws):

import fs from "node:fs";
import WebSocket from "ws";

const API_KEY = process.env.MODELGO_API_KEY;
const MODEL = "qwen3.8-omni-flash-realtime";
const url = "{{ ws_endpoint }}/api-ws/v1/realtime?model=" + MODEL;

const audio = fs.readFileSync("input.pcm"); // 16 kHz / 16-bit LE / mono
const out = fs.createWriteStream("reply.pcm"); // 24 kHz / 16-bit LE / mono
const ws = new WebSocket(url, { headers: { Authorization: `Bearer ${API_KEY}` } });

ws.on("message", (raw) => {
  const ev = JSON.parse(raw.toString());
  switch (ev.type) {
    case "session.created":
      ws.send(JSON.stringify({
        type: "session.update",
        session: {
          modalities: ["text", "audio"],
          turn_detection: null,
          audio: {
            input: { format: { type: "pcm", sample_rate: 16000 } },
            output: { format: { type: "pcm", sample_rate: 24000 } },
          },
        },
      }));
      break;
    case "session.updated":
      for (let i = 0; i < audio.length; i += 3200) {
        ws.send(JSON.stringify({
          type: "input_audio_buffer.append",
          audio: audio.subarray(i, i + 3200).toString("base64"),
        }));
      }
      ws.send(JSON.stringify({ type: "input_audio_buffer.commit" }));
      break;
    case "input_audio_buffer.committed":
      ws.send(JSON.stringify({ type: "response.create" }));
      break;
    case "response.audio.delta":
      out.write(Buffer.from(ev.delta, "base64"));
      break;
    case "response.audio_transcript.delta":
    case "response.text.delta":
      process.stdout.write(ev.delta);
      break;
    case "response.done":
      console.log("\nusage:", JSON.stringify(ev.response.usage));
      ws.close();
      out.end();
      break;
    case "error":
      console.error("error:", ev.error);
      ws.close();
      break;
  }
});
ws.on("error", console.error);

Wrap the returned reply.pcm (24 kHz, 16-bit, mono) into a playable WAV:

import wave
from pathlib import Path

with wave.open("reply.wav", "wb") as f:
    f.setnchannels(1)
    f.setsampwidth(2)
    f.setframerate(24000)
    f.writeframes(Path("reply.pcm").read_bytes())

To smoke-test connectivity without audio, send one conversation.item.create with an input_text content item, then response.create, to get a text-only reply.

Session configuration

Right after connecting, the server sends session.created (with defaults); the client overrides with session.update. Common fields:

The legacy input_audio_format / output_audio_format fields still work, but new integrations should use audio.input.format / audio.output.format.

Interruption

With a non-null turn_detection, the server VAD drives turns. When the user speaks again:

1. the server emits input_audio_buffer.speech_started;
2. the client stops playing buffered reply audio and sends response.cancel;
3. the server then emits response.done (status cancelled) and starts a new turn.

In manual mode (turn_detection: null) the client sends input_audio_buffer.commit and response.create after recording.

Billing

• Token-based, per modality: input is split into text/image/video and audio; output into text and audio. When the model produces speech, the audio and its transcript text are billed separately.
• Preauthorization at connect: a session-level estimate is held when the connection opens. If the balance cannot cover it, the handshake returns 402 and no connection is established.
• Settled after close on actual usage: the total of every response.done usage over the connection is settled at close — refund or top-up applies. A session that meters nothing releases the hold and charges nothing.

Check the console model page for current prices.

Limits

Next steps

• API reference: GET /api-ws/v1/realtime
• Authentication
• Errors and rate limits