Streaming

Set "stream": true in the request body. Streaming works in both inference formats and does not change pricing.

Anthropic emits events such as message_start, content_block_delta, and message_stop. OpenAI emits data: {...} chunks and finishes with data: [DONE].

from openai import OpenAI
client = OpenAI(api_key="mg_live_...", base_url="https://api.model-gate.com/v1")
stream = client.chat.completions.create(model="claude-opus-4.8", messages=[{"role":"user","content":"Hello"}], stream=True)
for chunk in stream:
    text = chunk.choices[0].delta.content
    if text:
        print(text, end="", flush=True)

Keep the connection open and disable reverse-proxy buffering. Closing the client connection cancels upstream generation; already generated usage remains billable. See Errors for retry guidance.

Completion and missing usage

HTTP 200 confirms the stream began, not that inference finished. A provider error or interrupted/truncated stream is shown as failed or incomplete in Usage. Chat Completions completion is a finish reason for every requested choice or [DONE]; Responses uses response.completed; Anthropic uses message_stop. Keep the request ID and treat already received text/tool arguments as partial. The gateway does not silently repeat this request.

Provider-reported usage, including zero or partial counters, is authoritative. A confirmed successful text stream without usage may receive an explicitly labelled estimate based on decoded text/reasoning/tool-argument delta bytes, excluding SSE/JSON framing; explicit output limits cap estimates only. Failed or unsupported streams without usage show Usage unknown · review required and do not receive an automatic byte-estimated charge. Unknown is not proof that the provider execution was free. See Errors.

Quiet periods and transport completion

Comment lines beginning with : may arrive while the provider is quiet. Ignore these as SSE keepalives: they contain no generated text or usage. Parse event boundaries rather than treating each network read as a token. LF, CRLF and CR line endings are supported.

Keep reading Chat Completions after finish_reason to receive any final usage and [DONE]. A valid [DONE], response.completed or message_stop ends its protocol response without waiting for the provider to close its connection. A stream idle/lifetime error is a failure, not a synthetic successful terminal. Retain the request ID and do not automatically replay a partial billable request.

A delayed initial HTTP response is separate from a pause inside an established SSE stream. Keepalives start only after the provider returns streaming response headers; they do not guarantee model output or bypass upstream HTTP errors. A provider that returns one complete content event cannot be made incremental by splitting its finished text in the client.