Showcase  /  Benchmark

SGLang started replying 153 ms sooner on long prompts with Affordance preparing them

When you send a conversation to a model server, the server has to turn it into tokens before the model can start. On a long conversation that step takes real time. We replaced it with Affordance inside SGLang, an open-source model server, and the wait for the first word of the reply fell from 1,121 ms to 968 ms on prompts over 120,000 tokens.

What a server does before the model starts

A model does not read text. It reads tokens, which are numbers that each stand for a piece of a word. A request arrives at the server as a list of messages, so the server has two jobs to do first.

The first job is to join the messages into one long string, in the exact format the model was trained on. The program that does this is called a chat template, and it is written in a template language called Jinja. The second job is to split that string into tokens. The program for that is called a tokenizer.

Both programs start from the first character on every request. An agent sends its whole conversation again each time it takes a step, with one new turn at the end. So at step 100 of a coding session, the server processes 99 turns that it has already processed 99 times.

What we changed

Affordance does both jobs in one step. It goes from the messages straight to tokens, and it keeps the turns it has already done, so a request costs only its new turn.

We put it into SGLang in place of the template and the tokenizer. The change is 140 lines in seven files. Before timing anything we checked that nothing else had changed: on 2,703 test requests, including ones with tools, SGLang with Affordance produced exactly the same tokens as SGLang without it.

The result on SGLang

We ran Qwen3.6-27B on one RTX PRO 6000 graphics card. One conversation grew by about 1,600 tokens a turn until it reached 164,000 tokens. On each request we timed how long the first word of the reply took to arrive.

Each bar is the mean of two runs. Both versions gave the same replies.

The gap is larger when the model's own work is small. That happens when you send the same long prompt again, to retry it or to get a second answer. The server remembers most of its work from the first time, so preparing the prompt is most of the wait.

Where the time went

We measured the two jobs by themselves, with no model running, on 200 prompts from a growing conversation. SGLang joined the messages into strings in 0.18 seconds, and then spent 17.6 seconds splitting those strings into tokens. Affordance did both jobs for the same 200 prompts in 0.47 seconds.

The same change in vLLM

vLLM is another open-source model server. We made the same change there, in 188 lines, and checked 2,880 requests for identical tokens. vLLM's tokenizer is much faster than SGLang's, and the wait for the first word did not change: 1,161 ms with Affordance and 1,158 ms without, on prompts over 120,000 tokens.

Both changes use the same Affordance apps for the model's format. We wrote those for 57 models and checked each one against the official template.