Category Archives: 000LLM

00 - Overview, 000LLM

Llm Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at LLM Overview.

So, what exactly is a Large Language Model? At its core, it’s a neural network that’s been trained on enormous amounts of text. We’re talking billions of web pages, books, articles, code repositories — you name it. The model learns statistical patterns across all of that data, and what makes it interesting is that after training, you can point it at all kinds of language tasks without having to retrain it for each one specifically.

Let’s talk about how one of these models actually processes your input, because understanding the flow helps a lot. You start with whatever text you type in — your prompt. That text gets broken down into tokens, which are small chunks of text, sometimes whole words, sometimes just parts of words. Those tokens then pass through a series of transformer layers, which use a mechanism called self-attention. Self-attention is what lets the model relate any token to any other token in the input — so it can understand context across the whole passage, not just word by word in sequence. After that, the model predicts the next token, and then the next, and the next — building up its response one piece at a time until it’s done. That overall design — the transformer architecture — was introduced in a 2017 paper and is the foundation of every major LLM you’ll encounter today.

Now let’s look at some key properties you’ll see referenced when people talk about these models. There are six worth knowing. First, parameters — these are the learned weights inside the model, and they’re what the model actually “knows.” Sizes range from about one billion parameters on the smaller end up to over a trillion for the largest models. Second, the context window — this is the maximum amount of text the model can hold in mind at once, measured in tokens. Older models handled around eight thousand tokens; newer ones go up to two million or more. Third, temperature — this controls how random or creative the outputs are. Set it to zero and you get very consistent, deterministic responses. Push it higher and the outputs get more varied and creative. Fourth and fifth are top-p and top-k, which are sampling filters — they give you finer control over how diverse the model’s word choices are. And sixth, RLHF — Reinforcement Learning from Human Feedback — which is a technique used to tune the model so it follows instructions well and behaves in line with human preferences. There’s also quantization, which is a compression technique that reduces the size of model weights so the model can run more efficiently.

Training one of these models happens in phases. The first phase is pre-training, where the model learns next-token prediction across that massive corpus of text — web content, books, code, all of it. The second phase is supervised fine-tuning, where the model is trained on curated question-and-answer pairs to get better at following instructions. The third phase involves RLHF or a similar technique called RLAIF, where a reward model and optimisation process align the model’s outputs more closely with what humans actually want. And finally, the model gets evaluated against benchmarks — standardised tests like MMLU for general knowledge, HumanEval for code, HellaSwag for commonsense reasoning, and MATH for mathematical problem-solving.

When it comes to what these models can actually do, the range is broad. They can generate text across formats — prose, poetry, code, structured data. They can summarise long documents down to the key points. They can classify and categorise content, including sentiment analysis. They can extract structured fields out of unstructured text, answer questions based on context you give them, and work through multi-step reasoning problems. They can write, explain, refactor, and debug code. They handle translation between languages. And increasingly, they can use tools — meaning they can call external APIs and functions as part of completing a task.

But there are real limitations you should keep in mind. Hallucination is probably the most important one — models can generate text that sounds completely plausible but is factually wrong, and they won’t always flag it. Second, there’s a knowledge cutoff — the model only knows what was in its training data, so recent events won’t be reflected. Third, the context limit means you can’t just feed it an arbitrarily large document and expect it to handle the whole thing. Fourth, models have no persistent memory by default — every time you start a new conversation, the model starts fresh, unless you’re using retrieval tools or external memory systems. And finally, models can be quite sensitive to how you phrase things — a small change in your prompt can meaningfully shift the output.

Those limitations aren’t reasons to avoid LLMs, but they’re important to design around. Knowing where the edges are is half the battle.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000LLM

Llm Wiki Schema

Welcome to the NZRT Wiki Podcast. Today we’re looking at LLM Wiki Schema.

This episode covers how notes are structured in the NZRT LLM knowledge vault. Whether you’re creating a new skill file or reviewing an existing one, this schema tells you exactly what to include and how to format it.

Let’s start with the frontmatter — that’s the block of metadata sitting at the very top of every skill note, before any main content. Think of it as the ID card for your document.

The frontmatter is written in a format called YAML, and it contains eight fields. Here’s what each one does.

First, tags. Required, and it holds a list of labels that make your note searchable and categorisable. Every note gets the vault-level tag 000LLM, the general LLM tag, a category tag, and a skill-specific slug. We’ll come back to those in a moment.

Second, aliases. Also required. These are human-readable names for the skill — alternative ways someone might refer to it, so if your file uses a technical slug, aliases give you the friendly labels people would actually search for.

Third, role. Required, and always set to the vault maintainer role, which in this case is xc.

Fourth, category. Required. This tells you which broad type the skill belongs to. There are seven options, and we’ll cover those shortly.

Fifth, complexity. Required. Low, medium, or high — reflecting how much effort the skill takes to implement.

Sixth, status. Required. A note moves through three stages: draft, review, and complete. This field tracks where it sits in that lifecycle.

Seventh, created. Required. The date the note was first made, in ISO 8601 format — that’s year, then month, then day.

Eighth, related. Optional. This lets you link out to other skill notes connected to this one.

Now let’s talk about the body of a skill file. Every note follows the same section order, and that consistency is what makes the vault navigable.

You start with the skill name as the top-level heading, followed by a single sentence defining what the skill is. Then comes an overview explaining what the skill is, what an LLM does when performing it, and why it matters. After that, a section called How It Works gives you a step-by-step or conceptual breakdown of the mechanism.

Next is Prompt Patterns — this is where reusable templates live. Placeholders in those templates are written in a specific style: all uppercase letters, words separated by underscores, and wrapped in double curly braces. So something like “document text” would appear as a placeholder in that format rather than as plain words.

Then you have an Examples section with concrete input and output pairs showing the skill in action, followed by NZRT Applications, which maps the skill to real work across WordPress, Dolibarr, Nextcloud, or TOGAF. After that comes Parameters and Tuning, covering things like temperature settings, token limits, and model recommendations. Then Limitations, documenting known failure modes and when not to use the skill. And finally, Related Skills, with links to connected notes and a one-line explanation of each relationship.

Now let’s cover the seven category types. Foundational covers core LLM techniques that everything else builds on — prompt engineering sits here. Text-processing is for skills that take input and transform it into a different form of text, such as summarization, extraction, and classification. Development covers code and technical artifact generation. Retrieval is about grounding LLM output in source documents — this is where RAG and question answering live. Analytical covers multi-step reasoning over complex inputs. Integration connects LLMs to external systems, like tool use and function calling. And architecture handles multi-LLM orchestration and agent design — things like pipelines and agent frameworks.

The tag taxonomy builds on all of this. At the top level you have three general tags: one for the vault itself, one for general LLM content, and one for broader AI and machine learning context. Then come the seven category tags matching the types we just covered. Each individual skill note also gets its own slug tag — examples include prompt-engineering, code-generation, summarization, data-extraction, question-answering, classification, reasoning, tool-use, rag, and agents. Finally, there are integration tags for the specific systems this vault connects to: wordpress, dolibarr, nextcloud, togaf, and nzrt.

The naming conventions are straightforward. Skill files live in a Skills folder and use lowercase words separated by hyphens, with a dot-md extension. Section headings use title case. And wikilinks use the slug form of the skill name.

Finally, the complexity guide gives you a clear way to decide which level applies. Low means a single prompt with no external dependencies that works reliably out of the box. Medium means you’ll need some prompt tuning, a few examples to guide the model, or some light post-processing of the output. High means you’re dealing with system design, external tools, retrieval pipelines, or multi-step chains.

That covers the full schema — frontmatter, body structure, categories, tags, naming conventions, and complexity levels. If you’re writing a new skill file for the vault, this is your complete checklist.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000LLM, 03 - Models & Providers

Models Providers

Welcome to the NZRT Wiki Podcast. Today we’re looking at Models & Providers.

If you’ve been working with AI tools at NZRT, you’ve probably noticed there’s no shortage of model options out there. This episode walks you through the main providers we work with, how to pick the right model for a given job, and what you can expect to pay.

Let’s start with Anthropic, the company behind Claude. There are three Claude models you’ll encounter. First is Claude Haiku 4.5 — this is your go-to for fast, high-volume work where cost matters. Think classification tasks, quick extractions, anything you need to run at scale. Then there’s Claude Sonnet 4.6, which sits in the middle and is probably the model you’ll reach for most often. It handles coding, reasoning, and writing really well without breaking the budget. At the top end is Claude Opus 4.7, the most capable of the three — best suited for complex reasoning tasks and working through long documents. All three Claude models support tool use, vision, extended thinking, computer use, and MCP integration. They all also carry a two-hundred-thousand token context window, which is substantial.

Next up is OpenAI’s GPT lineup. You have four key models here. GPT-4o is multimodal, fast, and broadly capable across a wide range of tasks. GPT-4o mini is the lighter, cheaper version — great for classification and anything you need done quickly at low cost. Then there are the reasoning-focused models: o1 uses extended thinking tokens to work through problems step by step, and o3 pushes that even further with more advanced reasoning and a two-hundred-thousand token context. GPT-4o and GPT-4o mini both sit at one-hundred-and-twenty-eight thousand tokens of context.

Moving on to Google’s Gemini family. This is where context windows get truly massive — all three Gemini models support up to one million tokens. Gemini 1.5 Pro is the standout for long-context multimodal work. Gemini 2.0 Flash gives you speed and low cost. And Gemini Ultra sits at the top for maximum capability. If you ever find yourself dealing with an enormous document — something north of five-hundred-thousand tokens — Gemini 1.5 Pro is where you want to be.

Now let’s talk about open-source and self-hosted options, because not every use case needs a cloud API. There are five models worth knowing here. Llama 3.3 from Meta at seventy billion parameters is a strong open-source baseline. Mistral Large at a hundred-and-twenty-three billion parameters is European-built and particularly solid for coding tasks. Qwen 2.5 from Alibaba comes in at seventy-two billion parameters and handles multilingual work well. DeepSeek R1 is a Chinese-built model using a mixture-of-experts architecture, weighing in at six-hundred-and-seventy-one billion parameters, with strong reasoning capability. And Phi-4 from Microsoft is a compact fourteen-billion-parameter model — efficient and well-suited for running on less powerful hardware. You can run all of these locally using tools like Ollama, LM Studio, vLLM, or llama.cpp.

So how do you choose? The wiki lays out a handy decision guide. For high-volume classification or extraction work, reach for Haiku 4.5 or GPT-4o mini. For code generation and general reasoning, Sonnet 4.6 or GPT-4o are your best bets. Complex architecture work or long documents point you toward Opus 4.7. If your context needs are enormous, Gemini 1.5 Pro handles that. Privacy-sensitive or on-premise requirements mean you’re looking at Llama 3.3 running locally via Ollama. And for extended step-by-step reasoning chains, o1, o3, or Claude with extended thinking enabled are your options.

On pricing — all the major APIs charge per token, and they bill input and output separately. The wiki groups these into three tiers. At the cheap end, models like Haiku and GPT-4o mini run somewhere between ten cents and forty cents per million tokens. The mid-tier — Sonnet and GPT-4o — sits between one and five dollars per million tokens. Premium models like Opus or o1 can run anywhere from fifteen to seventy-five dollars per million tokens. One thing worth knowing: both Claude and Gemini support prompt caching, which means repeated system prompts don’t cost you full price every time — a useful lever if you’re running the same setup across many requests.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000LLM, 02 - Prompt Engineering

Prompt Engineering

Welcome to the NZRT Wiki Podcast. Today we’re looking at Prompt Engineering.

So what exactly is prompt engineering? Put simply, it’s the practice of designing the inputs you give to a large language model so that you get accurate, useful, and consistent outputs back. Think of it as learning how to communicate with an AI in a way that actually works.

Let’s start with the core principles. There are five of them. First, be explicit — state your task, the format you want, and any constraints as clearly as possible. Second, provide context — include relevant background information right there in your prompt. Third, use examples — giving the model a few examples before your real question dramatically improves accuracy. Fourth, specify your output format — tell it whether you want JSON, a table, a bullet list, or plain prose. And fifth, constrain the scope — tell the model not just what to do, but what not to do.

Now let’s walk through the main prompting patterns.

The first is zero-shot prompting. No examples at all — you just rely on the model’s existing knowledge. Imagine asking it to classify the sentiment of a product review as positive, negative, or neutral. You give it the review text, and off it goes. Straightforward.

The second pattern is few-shot prompting. Here you provide two to five examples before your actual request. So you might show it that a review saying “Loved it!” is positive, “Broken on arrival” is negative, and “It’s okay” is neutral — then you present your real review and let the model follow the pattern. This significantly improves accuracy, especially on classification tasks.

Third is chain-of-thought prompting. Instead of asking for a direct answer, you ask the model to reason step by step before giving you a result. For example, you might give it a maths problem — a store has one hundred and twenty items, thirty percent are on sale, how many are full price — and ask it to work through each step. Showing its reasoning dramatically reduces errors.

Fourth is role prompting. You assign the model a persona to shape its response style. Something like telling it to act as a senior New Zealand tax accountant and answer in plain English. That framing steers the tone, vocabulary, and depth of everything that follows.

And fifth is structured output prompting. When you need machine-readable results, you just specify the format. You might say: return a JSON object with keys for name, date, amount, and currency. Clear, precise, done.

Now, when it comes to writing a system prompt, the order of your content matters. The recommended structure is: start with a role or persona, then describe the task, then list your constraints and rules, then specify your output format, and finish with any examples.

Let’s talk about what goes wrong. There are six common failure modes to be aware of.

The first is hallucination — the model invents facts to fill gaps. The fix is to ground it with retrieved context, a technique called Retrieval-Augmented Generation, or RAG.

The second is when the model ignores your instructions. This usually happens because your key rules got buried in a long prompt. Move them to the top or the very bottom where they carry more weight.

Third is verbose output — the model just goes on too long. Add an explicit length constraint, something like “be concise, maximum one hundred words.”

Fourth is wrong format — the model doesn’t give you what you asked for. Provide an exact format example so there’s no ambiguity.

Fifth is sycophancy — where the model agrees with incorrect statements just to please you. Counter it by explicitly telling it not to agree if something is wrong.

And sixth is over-refusal — the model refuses a perfectly valid request. This usually resolves when you rephrase and add context about your intent.

Finally, let’s look at some ideas from researcher Andrej Karpathy, who frames large language models as the core of a new kind of operating system. In this view, the model is like the CPU, and your prompt is the program running on it. The context window — everything the model can see at once — is like RAM, so you want to be deliberate about what you put in there. You can extend a model’s capabilities by giving it access to external tools. When you chain a model together with tools and memory in a loop, you get what’s called an agent. And here’s one of the most practical insights: verifying an answer is easier than generating one from scratch. That means you can use a language model to check its own outputs — a powerful technique worth building into your workflows wherever accuracy matters.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000LLM, 01 - Core Concepts

Tokens Context

Welcome to the NZRT Wiki Podcast. Today we’re looking at Tokens & Context.

Let’s start with the basics. What exactly is a token? A token is the fundamental unit of text that a large language model works with. Before a model can process anything, your text gets broken down into tokens — chunks that map to numbers in the model’s vocabulary. The process of doing this is called tokenization, and one popular method is called Byte Pair Encoding, or BPE for short.

So how big is a token in practice? Here are some useful rules of thumb. A single token is roughly four characters of English text, or about three quarters of a word. One hundred tokens works out to around seventy-five words. And a full page of text will typically cost you somewhere between five hundred and seven hundred and fifty tokens. One thing worth knowing: code, non-English languages, and special characters tend to use more tokens per character, so they’re less efficient from a token budget perspective.

Next up is the context window. This is the total number of tokens a model can handle in a single call, and that includes both what you send in and what the model sends back. Think of it as the model’s working memory. If you exceed the limit, the model simply cannot see what falls outside it.

Different models have very different context windows. Looking at some current examples: Claude Haiku, Sonnet, and Opus all offer two hundred thousand tokens. GPT-4o and Llama 3.3 70B each come in at one hundred and twenty-eight thousand. And Gemini 1.5 Pro sits at the top with a massive one million tokens. But bigger is not always better. Longer contexts increase both latency and cost, and not everything in a long context gets equal attention from the model.

That brings us to attention and position. The mechanism behind how models process context is called self-attention. It lets every token look at every other token in the context, but this gets computationally expensive fast. Specifically, it scales quadratically — so if you double the context length, you roughly quadruple the compute required.

Position also has a real effect on recall. Content placed at the very start or the very end of your prompt tends to be remembered most reliably. Anything buried deep in the middle is more likely to get overlooked. This is sometimes called the lost in the middle effect. The practical takeaway: put your most critical instructions at the beginning or end of your prompt, not sandwiched in the middle of a long document.

Now let’s talk about temperature and sampling — this is how you control the way a model generates output. Temperature is the most commonly used control. Set it to zero and the model becomes fully deterministic, always picking the most likely next token, giving you consistent repeatable output. Push it up to around zero point five and you get a balance of consistency with some variation. At one point zero you get the full sampling distribution, and above one the output becomes more creative and random, though it can start to lose coherence.

There are two other sampling controls worth knowing about. Top-p, sometimes called nucleus sampling, restricts the model to only sampling from tokens that together account for a certain share of the probability mass. Top-k does something similar but simply limits the choice to the most likely K tokens at each step. And max tokens is a hard cap — the model stops generating output at whatever number you set, regardless of whether it has finished.

The rule of thumb here is straightforward. If you need factual or structured output, keep temperature at zero or close to it. If you are brainstorming or want creative variety, something in the range of zero point seven to one point zero works well.

The last area to cover is how prompts are structured in a conversation. There are three roles. The system role is for instructions, persona setup, and constraints — it gets set at the start of a session and tells the model how to behave. The user role is for your actual input — questions, tasks, documents you want the model to work with. And the assistant role represents the model’s response, which can sometimes be pre-filled to guide the direction of the output.

One thing to keep in mind during multi-turn conversations is that all previous turns accumulate in context. Every message you send and every reply the model gives adds to the token count. Over a long session this eats into your token budget, so it is worth staying aware of how much context you are building up as a conversation grows.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000LLM, 04 - Tool Use & Agents

Tool Use Agents

Welcome to the NZRT Wiki Podcast. Today we’re looking at Tool Use & Agents.

Let’s start with the concept of tool use, sometimes called function calling. At its core, this is the ability for a large language model to reach out and interact with the world beyond its own knowledge. When you send a prompt to an AI, rather than just generating a text answer, the model can recognise that it needs to call an external tool to get the job done. It does this by producing a structured piece of data, essentially a description of what tool it wants to use and what inputs it needs. Your application picks that up, runs the actual tool, whether that’s an API call, a database query, or a file read, and then passes the result back to the model, which uses it to form its final response.

To make that concrete, imagine Claude deciding it needs to check the weather. It would output something that says: call the get weather function, with Auckland, New Zealand as the location. Your app runs that, gets the forecast, hands it back, and Claude says something sensible like “It looks like it’ll be cloudy in Auckland today.” The model never directly calls anything itself. It generates the instruction, and your host application does the actual work.

Next up is Retrieval-Augmented Generation, or RAG. This is a technique for grounding a model’s responses in real, up-to-date information rather than relying purely on what it learned during training. The idea is that when you ask a question, your system first converts that question into a mathematical representation called an embedding, then searches a database of similarly encoded document chunks to find the most relevant pieces of content. Those chunks get inserted into the prompt as context, and the model generates its answer based on that retrieved material rather than guessing.

There are a few moving parts to know about here. First, chunking: your source documents get split into overlapping segments, typically somewhere between two hundred and five hundred tokens each, so they’re a manageable size for retrieval. Second, an embedding model converts both your query and your document chunks into vectors, which are essentially lists of numbers that capture meaning. Models like text-embedding-3-small are commonly used for this. Third, a vector store holds all those embeddings and lets you search them by similarity. Popular options include Pinecone, Chroma, pgvector, Qdrant, and Meilisearch. And finally, a reranker can take the top results from that search and re-score them more carefully before they go into the prompt, improving quality even further.

Now let’s talk about agents. An agent is a large language model running in what’s called an agentic loop: it perceives a goal, reasons about what to do, takes an action using a tool, observes the result, and then repeats that cycle until the task is complete or it hands off to something else. This is what separates an agent from a simple chatbot. It’s not just answering your question once, it’s working through a problem across multiple steps.

There are several common patterns for how agents are designed. The ReAct pattern, short for Reason and Act, has the model interleave its thinking with its tool calls, reasoning out loud before each action. Plan-and-Execute agents generate a full plan upfront and then carry out each step in sequence. Reflection agents go a step further and critique their own output, retrying if something doesn’t look right. Multi-agent setups have an orchestrator model delegating subtasks to specialist subagents, each with their own focus area. And Human-in-the-Loop agents pause at key decision points and wait for a human to approve before continuing. Each pattern suits different use cases depending on how much autonomy and reliability you need.

One important standard in this space is MCP, the Model Context Protocol, released by Anthropic in 2024. MCP provides a unified way for language models to connect to tools and data sources. Think of it as a common language that lets Claude talk to a wide range of external systems without needing custom integration code for each one. In an MCP setup, you have the model itself, then an MCP host like Claude Code or Claude Desktop, and then MCP servers that expose the actual capabilities. Those servers can provide access to filesystems, databases, APIs, browsers, and more. MCP supports two transport modes: stdio for running things locally, and server-sent events over HTTP for remote connections. The things MCP servers can expose fall into three categories: tools, which are functions the model can call; resources, which are data the model can read like files or database rows; and prompts, which are reusable templates. MCP servers can be written in Python or TypeScript using the official SDK.

Finally, a quick word on memory. Language models are stateless by default, meaning each new conversation starts completely fresh. But there are patterns to work around this. You can keep prior turns or summaries in the prompt itself. You can store facts in a database and retrieve them by key. You can use semantic memory, embedding past interactions and retrieving them by similarity. You can log full conversation histories and summarise them on demand. Or you can encode persistent rules and persona directly into the system prompt, which is called procedural memory. Each approach has trade-offs in cost, speed, and fidelity, and many real-world systems combine several of them.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.