Welcome to the NZRT Wiki Podcast. Today we’re looking at Tokens & Context.

Let’s start with the basics. What exactly is a token? A token is the fundamental unit of text that a large language model works with. Before a model can process anything, your text gets broken down into tokens — chunks that map to numbers in the model’s vocabulary. The process of doing this is called tokenization, and one popular method is called Byte Pair Encoding, or BPE for short.

So how big is a token in practice? Here are some useful rules of thumb. A single token is roughly four characters of English text, or about three quarters of a word. One hundred tokens works out to around seventy-five words. And a full page of text will typically cost you somewhere between five hundred and seven hundred and fifty tokens. One thing worth knowing: code, non-English languages, and special characters tend to use more tokens per character, so they’re less efficient from a token budget perspective.

Next up is the context window. This is the total number of tokens a model can handle in a single call, and that includes both what you send in and what the model sends back. Think of it as the model’s working memory. If you exceed the limit, the model simply cannot see what falls outside it.

Different models have very different context windows. Looking at some current examples: Claude Haiku, Sonnet, and Opus all offer two hundred thousand tokens. GPT-4o and Llama 3.3 70B each come in at one hundred and twenty-eight thousand. And Gemini 1.5 Pro sits at the top with a massive one million tokens. But bigger is not always better. Longer contexts increase both latency and cost, and not everything in a long context gets equal attention from the model.

That brings us to attention and position. The mechanism behind how models process context is called self-attention. It lets every token look at every other token in the context, but this gets computationally expensive fast. Specifically, it scales quadratically — so if you double the context length, you roughly quadruple the compute required.

Position also has a real effect on recall. Content placed at the very start or the very end of your prompt tends to be remembered most reliably. Anything buried deep in the middle is more likely to get overlooked. This is sometimes called the lost in the middle effect. The practical takeaway: put your most critical instructions at the beginning or end of your prompt, not sandwiched in the middle of a long document.

Now let’s talk about temperature and sampling — this is how you control the way a model generates output. Temperature is the most commonly used control. Set it to zero and the model becomes fully deterministic, always picking the most likely next token, giving you consistent repeatable output. Push it up to around zero point five and you get a balance of consistency with some variation. At one point zero you get the full sampling distribution, and above one the output becomes more creative and random, though it can start to lose coherence.

There are two other sampling controls worth knowing about. Top-p, sometimes called nucleus sampling, restricts the model to only sampling from tokens that together account for a certain share of the probability mass. Top-k does something similar but simply limits the choice to the most likely K tokens at each step. And max tokens is a hard cap — the model stops generating output at whatever number you set, regardless of whether it has finished.

The rule of thumb here is straightforward. If you need factual or structured output, keep temperature at zero or close to it. If you are brainstorming or want creative variety, something in the range of zero point seven to one point zero works well.

The last area to cover is how prompts are structured in a conversation. There are three roles. The system role is for instructions, persona setup, and constraints — it gets set at the start of a session and tells the model how to behave. The user role is for your actual input — questions, tasks, documents you want the model to work with. And the assistant role represents the model’s response, which can sometimes be pre-filled to guide the direction of the output.

One thing to keep in mind during multi-turn conversations is that all previous turns accumulate in context. Every message you send and every reply the model gives adds to the token count. Over a long session this eats into your token budget, so it is worth staying aware of how much context you are building up as a conversation grows.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.