affordance

Showcase  /  Benchmark

We trained a model with reinforcement learning on an Affordance app, from 40% to 77%, on exactly the tokens it saw

Reinforcement learning lets a model get better at a task by trying it many times and learning from the attempts that worked. We used it to train Qwen3-4B on a small Affordance app, and it went from 40.2% right answers to 77.3% on episodes it had never seen.

The part we cared most about is the training data. Most pipelines keep a conversation as a list of messages and turn it back into tokens with the model's chat template. On this task that round trip did not give back what the model wrote: after training it changed 261 of the model's 2,729 replies, and 297 of 1,766 before training. Almost all of the changes are whitespace, the newlines around a tool call.

With Affordance there is no round trip. Every one of the 345,983 model calls the model trained on matched, token for token, what the inference server had actually processed.

Reinforcement learning with a checkable answer

The method is often called RLVR, reinforcement learning with verifiable rewards. You need a task where a program can check the answer. The model attempts it many times. Attempts that got the right answer are rewarded, and training makes the steps the model took in those attempts more likely next time.

Training works on tokens, the pieces of text a model reads and writes. To make a step more likely, the trainer has to know exactly which tokens the model was shown and exactly which tokens it wrote. If either is slightly off, the trainer adjusts the model for something it did not do.

The chat template does not round-trip

Most agent frameworks store a conversation as a list of messages: the user's text, the model's reply, the tool call it made. To build training data, the trainer turns that list back into tokens with the model's chat template. We did that with this task's replies and compared the result with the tokens the model actually wrote.

Replies rebuilt with the chat templateRepliesChanged
Before training1,766297 (17%)
After 50 training steps2,729261 (10%)
Affordance's export, every call of both training runs345,9830

We read 81 of the changed replies. In 80 the only difference was whitespace. The model put a blank line before its tool call, and the template writes a single newline:

what the model wrote
 will now submit this as the largest number. \n\n

<tool_call>
{"name": "
what the template rebuilds
 will now submit this as the largest number.\n
<tool_call>
{"name": "note

The template is not wrong about the format. It parses the tool call out of the reply and writes it back the canonical way, and what the model actually wrote is lost. It looks harmless, but it changes what the trainer learns. Read in the rebuilt context, the token the model actually sampled there became about 23 nats less likely, a factor of roughly 14 billion. 60 of the 113 affected calls had a token move by more than 5 nats. A trainer working from rebuilt history is training on a conversation the model never had.

Decoding the model's tokens to text and encoding them again also changes 15 of the 1,766 replies: the same text, split into different tokens.

The same round trip costs time at inference. An agent that sends its history back as messages gets the rebuilt reply in its next prompt, so the server's cached copy of what the model wrote stops matching at the first changed token, and everything after it is computed again.

The task is an app

The model is shown a notebook of 16 to 48 four-digit numbers, but it can only see about 40 tokens of it at a time, a few rows. It has three tools: scroll the notebook, write a note to itself, and submit an answer. The reward is 1 if it submits the largest number in the notebook and 0 otherwise.

The task is easy to state and easy to fail. Before training, the model often submitted the largest number it had seen so far without scrolling to the end. It reached the last row in only 47% of episodes.

In Affordance the environment is an app like any other. The tools are functions, and the screen is a box with a 40-token limit:

notebook.tsxlines 17 to 32, lines 37 to 46
/** Scroll the notebook by a number of rows (negative scrolls up). */
function scroll({ by }: { by: number }) {
  const next = Math.min(Math.max(0, offset + Math.trunc(by)), numbers.length - 1);
  setOffset(next);
  return `The notebook now starts at row ${next}.`;
}
/** Replace your notes with new text. */
function note({ text }: { text: string }) {
  setNotes(text);
  return "Noted.";
}
/** Submit your final answer: the largest number in the notebook. This ends the task. */
function submit({ answer }: { answer: number }) {
  harness.stop({ answer });
  return "Submitted.";
}
...export function Screen({ numbers, offset, notes, window, fns }) {
  return <>
    <box label="notebook" max_tokens={window} more="...">
      {`Notebook (${numbers.length} rows), starting at row ${offset}:\n`}
      {numbers.slice(offset).map((n, i) => `${offset + i}: ${n}\n`)}
    </box>
    {`\nYour notes: ${notes || "(empty)"}\n`}
    {fns.map(fn => <Tool fn={fn}/>)}
  </>;
}

Exporting what the model actually saw

Affordance builds every prompt itself, so it knows the exact token ids at every step. Each new prompt in this app is the previous prompt with the model's reply and the app's response added to the end. So a whole episode is one sequence of tokens. The model's own tokens are trained on, and the tokens the app wrote are not.

written by the model: trainedwritten by the app (task, screens, tool results): not trained

One real episode from the final evaluation: 1,146 tokens, 7 model calls, a 20-row notebook, answered correctly.

To check the export, we recorded every request to the inference server and its response, byte for byte. Then we compared every exported episode against those recordings.

Checked against what SGLang received and returnedCount
Episodes34,304
Model calls345,983
Prompt tokens300,127,345
Tokens the model wrote13,771,592
Differences0

The training run

Each step, the model attempted 32 notebooks 8 times each. Within each group of 8, an attempt that did better than the group's average was made more likely, and one that did worse was made less likely. This is GRPO. Then the new weights were loaded into the inference server and the next step began.

We ran 70 steps, 17,920 episodes, in 45 minutes: two GPUs trained the model and two served it with SGLang. Every 10 steps we measured it on the same 256 held-out episodes.

Share of episodes answered correctly, by training step

Held-out: the same 256 episodes every 10 steps, on notebooks never trained on. Training: each step's own 256 episodes.

held-out episodestraining episodes
0%25%50%75%100%010203040506070 training step Step 0: 31.3% of training episodes rewardedStep 1: 42.2% of training episodes rewardedStep 2: 37.1% of training episodes rewardedStep 3: 49.6% of training episodes rewardedStep 4: 52.7% of training episodes rewardedStep 5: 43.4% of training episodes rewardedStep 6: 51.2% of training episodes rewardedStep 7: 52.7% of training episodes rewardedStep 8: 51.6% of training episodes rewardedStep 9: 50.0% of training episodes rewardedStep 10: 51.2% of training episodes rewardedStep 11: 39.8% of training episodes rewardedStep 12: 65.6% of training episodes rewardedStep 13: 62.5% of training episodes rewardedStep 14: 48.4% of training episodes rewardedStep 15: 53.9% of training episodes rewardedStep 16: 38.3% of training episodes rewardedStep 17: 41.8% of training episodes rewardedStep 18: 50.4% of training episodes rewardedStep 19: 47.3% of training episodes rewardedStep 20: 64.8% of training episodes rewardedStep 21: 64.8% of training episodes rewardedStep 22: 60.9% of training episodes rewardedStep 23: 72.3% of training episodes rewardedStep 24: 73.0% of training episodes rewardedStep 25: 60.5% of training episodes rewardedStep 26: 78.1% of training episodes rewardedStep 27: 78.9% of training episodes rewardedStep 28: 84.0% of training episodes rewardedStep 29: 83.2% of training episodes rewardedStep 30: 81.3% of training episodes rewardedStep 31: 84.0% of training episodes rewardedStep 32: 76.6% of training episodes rewardedStep 33: 84.4% of training episodes rewardedStep 34: 82.8% of training episodes rewardedStep 35: 84.0% of training episodes rewardedStep 36: 75.0% of training episodes rewardedStep 37: 86.3% of training episodes rewardedStep 38: 74.6% of training episodes rewardedStep 39: 84.8% of training episodes rewardedStep 40: 78.1% of training episodes rewardedStep 41: 82.8% of training episodes rewardedStep 42: 73.0% of training episodes rewardedStep 43: 65.2% of training episodes rewardedStep 44: 87.5% of training episodes rewardedStep 45: 92.6% of training episodes rewardedStep 46: 93.8% of training episodes rewardedStep 47: 78.1% of training episodes rewardedStep 48: 75.8% of training episodes rewardedStep 49: 80.5% of training episodes rewardedStep 50: 77.3% of training episodes rewardedStep 51: 88.7% of training episodes rewardedStep 52: 89.5% of training episodes rewardedStep 53: 81.3% of training episodes rewardedStep 54: 77.3% of training episodes rewardedStep 55: 81.3% of training episodes rewardedStep 56: 93.8% of training episodes rewardedStep 57: 81.3% of training episodes rewardedStep 58: 78.1% of training episodes rewardedStep 59: 67.6% of training episodes rewardedStep 60: 81.3% of training episodes rewardedStep 61: 87.5% of training episodes rewardedStep 62: 81.3% of training episodes rewardedStep 63: 81.3% of training episodes rewardedStep 64: 86.7% of training episodes rewardedStep 65: 87.1% of training episodes rewardedStep 66: 78.1% of training episodes rewardedStep 67: 87.5% of training episodes rewardedStep 68: 84.4% of training episodes rewardedStep 69: 87.5% of training episodes rewarded Step 0: 40.2% right on held-out episodes (103 of 256)Step 10: 48.4% right on held-out episodes (124 of 256)Step 20: 56.3% right on held-out episodes (144 of 256)Step 30: 75.8% right on held-out episodes (194 of 256)Step 40: 77.3% right on held-out episodes (198 of 256)Step 50: 77.3% right on held-out episodes (198 of 256)Step 60: 77.3% right on held-out episodes (198 of 256)Step 70: 77.3% right on held-out episodes (198 of 256) held-out 77.3% 40.2%

What the model learned is the obvious missing habit. The share of episodes in which it scrolled to the last row before answering went from 47% to 100%. On a second set of new notebooks it went from 37.5% to 80.5%.

Training did not drift the model away from what the server sampled. At the same weights, the trainer's probabilities for the sampled tokens differed from SGLang's by 0.0059 nats per token on average.

One run on one machine, with a small model and a toy task, and no repeated seeds. Accuracy stopped rising after 40 steps, once the model answered each notebook the same way every time. The trained model scrolls in small steps, so on 56-row notebooks it needs more than the 16 turns it trained with: with 32 turns it scores 71.5%, against 25.4% before training.

Try it

The environment is the app above, run with affordance run against SGLang through Affordance's token-id binding, with every request recorded by setting AFFORDANCE_CAPTURE_DIR. The trainer is about 350 lines of PyTorch and is not published yet.