Welcome to the NZRT Wiki Podcast. Today we’re looking at Logging.

Let’s start with the big picture. In Kubernetes, when your containers run, they write output to two standard streams — standard output and standard error. That output doesn’t just disappear. The kubelet, which is the agent running on each node, captures it and writes it to the node’s file system. From there, a log shipper picks it up and sends it on to a centralised log storage system called Loki. And then Grafana sits on top of Loki so you can actually query and visualise everything. That’s the full chain — container output, to kubelet, to log shipper, to Loki, to Grafana.

Now let’s talk about how you access logs directly using kubectl. This is your first line of investigation when something looks wrong.

The most basic command gets the current logs from a pod. You specify the pod name and tell it which namespace to look in — in NZRT’s case that’s nzrt-prod. This gives you a snapshot of what that pod has written to its output streams up to this point.

If you want to watch logs as they come in live, you can follow them. This keeps the stream open and prints new lines as they arrive — useful when you’re actively watching a deployment or waiting for a specific event to show up.

Now, what if a pod has crashed and restarted? That’s where the previous flag comes in. When a container crashes and Kubernetes restarts it, the logs from the crashed container are gone from the current session — but you can still retrieve them by asking for the previous instance. This is often the most useful thing to look at when you’re debugging a crash.

If you’re working with a pod that runs more than one container inside it, you need to also specify which container you want logs from. You give both the pod name and the container name, and kubectl knows exactly where to look.

And finally, if you don’t know the exact pod name but you know the application label, you can ask for logs from all pods matching that label at once. For example, you could ask for all pods labelled as WordPress in nzrt-prod, and you’ll get output from every one of them in one go.

Now let’s go deeper into the log architecture — the pipeline that makes centralised logging work.

First, your pod writes to standard output or standard error. The kubelet on that node picks it up and writes it to a path on the node’s local file system, under a directory called var log pods.

From there, a DaemonSet runs on every node in the cluster. A DaemonSet means one copy of a pod runs on every single node automatically. At NZRT, that’s either Promtail or Grafana Alloy — both are log shippers that read the files the kubelet wrote and forward them on.

The destination is Loki, a log storage system designed specifically for Kubernetes environments. Loki stores your logs and indexes them by labels like namespace and app name, rather than indexing the full text of every log line. This keeps storage lean and efficient.

Finally, Grafana sits in front of Loki. This is where you go to actually query your logs, build dashboards, or set up alerts based on log content.

So the flow is: pod, to kubelet, to Promtail or Alloy, to Loki, to Grafana. Each step has a clear role.

Once you’re in Grafana, you query Loki using a language called LogQL. Let me walk you through a few examples of what those queries do in plain terms.

The most basic query selects all logs from a specific namespace and application — for example, show me everything from the nzrt-prod namespace where the app is WordPress.

You can filter further by looking for specific text in the log lines. For example, you might ask for all logs in the nzrt-prod namespace that contain the word error somewhere in the line. That’s a simple text filter and a quick way to spot problems.

If your logs are structured as JSON, which is common with modern applications, you can parse them and then filter on specific fields. So you could ask for logs where the parsed level field equals error — which is more precise than just searching for the word error anywhere in the raw text.

And if you want to understand log volume rather than read individual lines, you can use a rate query. This tells you how many log lines are being produced per second or per minute, averaged over a time window like five minutes. This is useful for spotting sudden spikes in log output, which often signals something going wrong upstream.

That’s the core of Kubernetes logging at NZRT — kubectl for direct pod access, a DaemonSet-based pipeline to ship logs into Loki, and Grafana with LogQL for querying and visualising everything. If you want to go further, check out the Monitoring Overview and the kubectl Reference pages in the wiki.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.