Understanding LLM Tokens: A Visual Playground

When you send a message to an LLM like Claude or ChatGPT, the model does not see your text. It sees tokens. I wanted to see them too, so I built a small page for that.

Try it here: LLM Tokens Playground

What Are Tokens?

Tokens are the units an LLM works with. When you send "How are you today?", the model gets:

• Token strings: ["How"," are"," you"," today","?"]
• Token IDs: [5299, 553, 481, 4044, 30]

"are" includes the space before it. Tokens usually carry their leading space and punctuation.

Why Tokens Matter

The model converts everything to token IDs, integers that stand for pieces of text, and answers in token IDs that get converted back to text. That is why context limits are in tokens, like 200k, and not in characters, and why the cost of a request is counted in tokens. Some odd behavior with numbers or spelling also becomes clearer once you see where the splits are.

The Playground

The LLM Tokens Playground shows for any text you type:

• The token strings
• The token IDs
• A table mapping each ID to its text fragment

It uses the o200k_base tokenizer, the one behind many current LLMs.

What I noticed

English and German short sentences cost about the same: "How are you today?" is 5 tokens, "Wie geht es dir heute?" is 6. Longer German words are a different story. "software developer" is 2 tokens, "Softwareentwickler" is 3, and "Donaudampfschifffahrtsgesellschaft" is 10: D, ona, ud, amp, fsch, if, ff, ahr, ts, gesellschaft. The tokenizer was trained on mostly English text, so it knows "gesellschaft" as one piece, but has to assemble the rest from fragments.

Numbers get split into groups of three digits: "1234567890" is 123, 456, 789, 0. And my own domain, "stefanwille.com", is 5 tokens, with ".com" as one of them.

The playground is open source and built with Next.js and the gpt-tokenizer library.