Welcome to the NZRT Wiki Podcast. Today we’re looking at Logging.

Let’s start with the big picture. In Kubernetes, when your containers run, they write output to two standard streams — standard output and standard error. That output doesn’t just disappear. The kubelet, which is the agent running on each node, captures it and writes it to the node’s file system. From there, a log shipper picks it up and sends it on to a centralised log storage system called Loki. And then Grafana sits on top of Loki so you can actually query and visualise everything. That’s the full chain — container output, to kubelet, to log shipper, to Loki, to Grafana.

Now let’s talk about how you access logs directly using kubectl. This is your first line of investigation when something looks wrong.

The most basic command gets the current logs from a pod. You specify the pod name and tell it which namespace to look in — in NZRT’s case that’s nzrt-prod. This gives you a snapshot of what that pod has written to its output streams up to this point.

If you want to watch logs as they come in live, you can follow them. This keeps the stream open and prints new lines as they arrive — useful when you’re actively watching a deployment or waiting for a specific event to show up.

Now, what if a pod has crashed and restarted? That’s where the previous flag comes in. When a container crashes and Kubernetes restarts it, the logs from the crashed container are gone from the current session — but you can still retrieve them by asking for the previous instance. This is often the most useful thing to look at when you’re debugging a crash.

If you’re working with a pod that runs more than one container inside it, you need to also specify which container you want logs from. You give both the pod name and the container name, and kubectl knows exactly where to look.

And finally, if you don’t know the exact pod name but you know the application label, you can ask for logs from all pods matching that label at once. For example, you could ask for all pods labelled as WordPress in nzrt-prod, and you’ll get output from every one of them in one go.

Now let’s go deeper into the log architecture — the pipeline that makes centralised logging work.

First, your pod writes to standard output or standard error. The kubelet on that node picks it up and writes it to a path on the node’s local file system, under a directory called var log pods.

From there, a DaemonSet runs on every node in the cluster. A DaemonSet means one copy of a pod runs on every single node automatically. At NZRT, that’s either Promtail or Grafana Alloy — both are log shippers that read the files the kubelet wrote and forward them on.

The destination is Loki, a log storage system designed specifically for Kubernetes environments. Loki stores your logs and indexes them by labels like namespace and app name, rather than indexing the full text of every log line. This keeps storage lean and efficient.

Finally, Grafana sits in front of Loki. This is where you go to actually query your logs, build dashboards, or set up alerts based on log content.

So the flow is: pod, to kubelet, to Promtail or Alloy, to Loki, to Grafana. Each step has a clear role.

Once you’re in Grafana, you query Loki using a language called LogQL. Let me walk you through a few examples of what those queries do in plain terms.

The most basic query selects all logs from a specific namespace and application — for example, show me everything from the nzrt-prod namespace where the app is WordPress.

You can filter further by looking for specific text in the log lines. For example, you might ask for all logs in the nzrt-prod namespace that contain the word error somewhere in the line. That’s a simple text filter and a quick way to spot problems.

If your logs are structured as JSON, which is common with modern applications, you can parse them and then filter on specific fields. So you could ask for logs where the parsed level field equals error — which is more precise than just searching for the word error anywhere in the raw text.

And if you want to understand log volume rather than read individual lines, you can use a rate query. This tells you how many log lines are being produced per second or per minute, averaged over a time window like five minutes. This is useful for spotting sudden spikes in log output, which often signals something going wrong upstream.

That’s the core of Kubernetes logging at NZRT — kubectl for direct pod access, a DaemonSet-based pipeline to ship logs into Loki, and Grafana with LogQL for querying and visualising everything. If you want to go further, check out the Monitoring Overview and the kubectl Reference pages in the wiki.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

Welcome to the NZRT Wiki Podcast. Today we’re looking at Monitoring Overview.

If you’re running workloads on NZRT’s Kubernetes cluster, you need to know how the observability stack is put together. Observability is really just a fancy word for being able to see what’s happening inside your systems — measuring performance, viewing logs, and checking whether your services are healthy. Today we’ll walk through the tools involved, how health probes work, and some quick commands you can use to check on things at any time.

Let’s start with the tools. NZRT uses six components to cover observability end to end. First, there’s Prometheus, which handles metrics scraping and storage — think of it as the thing that constantly collects numbers from your cluster, like CPU usage, memory, and request counts. Second is Grafana, which sits on top of Prometheus and turns all those numbers into dashboards and alerts you can actually read. Third is Loki, which handles log aggregation — it gathers log output from across your cluster so you can search and query it in one place. Fourth is Promtail, also known as Alloy, which is the log shipper that runs at the node level and feeds data into Loki. Fifth is kube-state-metrics, which exposes cluster state as metrics — things like whether a deployment has the right number of replicas, or whether a pod is pending. And sixth is metrics-server, which powers the kubectl top command so you can see live resource usage on pods and nodes.

Now, five of those six tools — Prometheus, Grafana, Loki, Promtail, and kube-state-metrics — all live in a namespace called monitoring. The last one, metrics-server, lives in kube-system, which is the core Kubernetes namespace reserved for system components.

Next up: health probes. These are a critical part of running production containers, and NZRT requires you to add them to every production container you deploy. There are two types, and they serve different purposes.

The first is a liveness probe. This is Kubernetes asking the question: is this container still alive and worth keeping around? In the example configuration, the liveness probe makes an HTTP GET request to the slash health path on port 80. It waits 30 seconds after the container starts before making its first check, then checks every 10 seconds after that. If the container fails to respond three times in a row, Kubernetes will restart it.

The second is a readiness probe. This one asks a different question: is this container ready to receive traffic? It’s similar in setup — it checks a path called slash ready on port 80 — but it starts checking sooner, after just 10 seconds, and checks more frequently, every 5 seconds. The failure threshold is still three. The key difference is what happens on failure: a failed readiness probe doesn’t restart the container, it just removes it from the load balancer rotation until it recovers. That’s an important distinction. Liveness failures restart. Readiness failures pause traffic.

Together, these two probes give Kubernetes the information it needs to manage your containers intelligently without any manual intervention from you.

Now let’s talk about quick checks — commands you can run at any time to get a fast read on what’s happening in the cluster.

The first command shows you resource usage across all pods in the production namespace. You’ll see each pod’s CPU and memory consumption at a glance. The second command is similar but zooms out to the node level, showing you how much each underlying machine is using overall.

The third command pulls recent events from the production namespace and sorts them by timestamp, so the most recent things that happened in the cluster are at the bottom. This is a great first stop when something seems off — events will often tell you about failed pulls, scheduling issues, or restarts before you even go looking.

The fourth command lets you describe a specific pod by name. When you run it, pay particular attention to two sections in the output: Conditions and Events. Conditions tell you the current state of the pod — whether it’s initialized, ready, and scheduled. Events tell you the history of what’s happened to it, including any probe failures, restarts, or errors. Between those two sections, you can usually diagnose most common pod issues without needing to dig any further.

If you want to go deeper from here, the wiki also covers Logging in its own article, and there’s a Cluster Overview page that gives you the broader picture of how the NZRT Kubernetes environment is structured. Both are worth reading alongside this one.

To summarise: NZRT’s monitoring stack uses Prometheus for metrics, Grafana for dashboards, Loki for logs, and a set of supporting tools to make all of it work. Every production container gets a liveness probe and a readiness probe. And when something goes wrong, you’ve got four quick commands to start your investigation — pod resource usage, node resource usage, recent events, and pod describe.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.