Reading model replies

Contract authority: public contracts, language and harness.

Rendering turns state into a model's tokens; reply parsing turns the model's tokens back into thinking, content and tool calls. Every chat view in views/index.json has a row in views/reply_protocols.json that names its reply syntax, the tokens that end a reply, and whether the generation prompt ends inside an open thinking block. The parsers live in src/reply.rs (Syntax, parse_reply_typed, Problem) and src/reply_formats.rs.

Every parser reports the same Problems: no_action, multiple_actions, prose_before_action, text_after_action, unclosed_think, bad_syntax, unknown_action, bad_args, cut_off. A reply cut off before its thinking closes is thinking (unclosed_think + cut_off), never content; a call cut off mid-way is never returned. Syntaxes whose values arrive as text (xml, glm, atem, llama4) are typed by the action's JSON schema ("3" → 3; Python literals such as {'a': True} are read too).

Decode replies with special tokens kept (decode(ids, false)): several families write their markers as special tokens (Mistral [TOOL_CALLS], gpt-oss channels, Glimmer headers, Llama <|python_tag|>, DeepSeek V3.2 ｜DSML｜).

#Adding a family

1. Write the view (views/<family>/…) and add it to views/index.json (conformance as usual).

2. Add its row to views/reply_protocols.json: syntax (an existing one, or null with a reason), stop (the token strings that end a reply), in_think (does the default generation prompt end inside an open thinking block?).

3. If the format is new, add a Syntax variant (src/reply.rs, name in Syntax::NAMES / Syntax::parse) and its parser in src/reply_formats.rs returning Parts.

4. Run cargo test --release --test reply_coverage. It fails for a view without a row, and it round-trips the parser through the view: it renders a conversation without and with an assistant turn holding tool calls (one call, two calls, string arguments, with reasoning) and parses what the template wrote for the model's turn. Where history is rendered differently from a fresh generation, the calls are checked in their history form. A rendering that cannot be read back for a template-specific reason goes in the test's KNOWN_LOSSY list with that reason.

5. To run the family as an agent, its template app (views/template_apps/) declares the same protocol in a zero-token <span label="protocol" syntax=… stop={[…]} in_think={…}/> (checked against the table by reply_coverage), passes the family's special tokens (bos_token, …), and calls templateInput(children, system, true, {alternate}).

   • Generated turns. The view renders a generated assistant turn (m.ids, with the turn's calls as tool_calls metadata) as exactly what its generation prompt was at that point, then tokens(m.ids), then the turn end only if the ids lack it. It updates its own state as after an assistant turn (DeepSeek's flags after tool output, gpt-oss's last tool). Every chat view does this. m.ids is a token run that reads like a list of ids (HARNESS_GUIDE section 5).
   • Laid-out content. A screen is a fragment: string operations on content (.includes, .split, iterating it as parts) must let fragments through as text.
   • alternate. For a format that requires alternating roles (Llama 2, Mistral v0.3), or places tools by the last user turn (Mistral Small 3.2): a user turn right after another user turn or a tool result joins it.
   • system null. For a format that moves the system prompt every turn (Mistral v0.3 puts it in the final user turn): no system turn.

   tests/template_sweep.rs runs every chat view this way, through a real agent, for three turns (a tool loop, or a text loop for a family without tool calls). It checks that each reply parses, that each prompt is an exact append of the last plus the reply, and that the turn boundary after a reply is the template's own (no doubled or missing stop). It reports the configuration each family needs.

Source: docs/REPLY_PARSING.md, about 4 minutes to read.