Welcome to the NZRT Wiki Podcast. Today we’re looking at Monitoring Overview.

If you’re running workloads on NZRT’s Kubernetes cluster, you need to know how the observability stack is put together. Observability is really just a fancy word for being able to see what’s happening inside your systems — measuring performance, viewing logs, and checking whether your services are healthy. Today we’ll walk through the tools involved, how health probes work, and some quick commands you can use to check on things at any time.

Let’s start with the tools. NZRT uses six components to cover observability end to end. First, there’s Prometheus, which handles metrics scraping and storage — think of it as the thing that constantly collects numbers from your cluster, like CPU usage, memory, and request counts. Second is Grafana, which sits on top of Prometheus and turns all those numbers into dashboards and alerts you can actually read. Third is Loki, which handles log aggregation — it gathers log output from across your cluster so you can search and query it in one place. Fourth is Promtail, also known as Alloy, which is the log shipper that runs at the node level and feeds data into Loki. Fifth is kube-state-metrics, which exposes cluster state as metrics — things like whether a deployment has the right number of replicas, or whether a pod is pending. And sixth is metrics-server, which powers the kubectl top command so you can see live resource usage on pods and nodes.

Now, five of those six tools — Prometheus, Grafana, Loki, Promtail, and kube-state-metrics — all live in a namespace called monitoring. The last one, metrics-server, lives in kube-system, which is the core Kubernetes namespace reserved for system components.

Next up: health probes. These are a critical part of running production containers, and NZRT requires you to add them to every production container you deploy. There are two types, and they serve different purposes.

The first is a liveness probe. This is Kubernetes asking the question: is this container still alive and worth keeping around? In the example configuration, the liveness probe makes an HTTP GET request to the slash health path on port 80. It waits 30 seconds after the container starts before making its first check, then checks every 10 seconds after that. If the container fails to respond three times in a row, Kubernetes will restart it.

The second is a readiness probe. This one asks a different question: is this container ready to receive traffic? It’s similar in setup — it checks a path called slash ready on port 80 — but it starts checking sooner, after just 10 seconds, and checks more frequently, every 5 seconds. The failure threshold is still three. The key difference is what happens on failure: a failed readiness probe doesn’t restart the container, it just removes it from the load balancer rotation until it recovers. That’s an important distinction. Liveness failures restart. Readiness failures pause traffic.

Together, these two probes give Kubernetes the information it needs to manage your containers intelligently without any manual intervention from you.

Now let’s talk about quick checks — commands you can run at any time to get a fast read on what’s happening in the cluster.

The first command shows you resource usage across all pods in the production namespace. You’ll see each pod’s CPU and memory consumption at a glance. The second command is similar but zooms out to the node level, showing you how much each underlying machine is using overall.

The third command pulls recent events from the production namespace and sorts them by timestamp, so the most recent things that happened in the cluster are at the bottom. This is a great first stop when something seems off — events will often tell you about failed pulls, scheduling issues, or restarts before you even go looking.

The fourth command lets you describe a specific pod by name. When you run it, pay particular attention to two sections in the output: Conditions and Events. Conditions tell you the current state of the pod — whether it’s initialized, ready, and scheduled. Events tell you the history of what’s happened to it, including any probe failures, restarts, or errors. Between those two sections, you can usually diagnose most common pod issues without needing to dig any further.

If you want to go deeper from here, the wiki also covers Logging in its own article, and there’s a Cluster Overview page that gives you the broader picture of how the NZRT Kubernetes environment is structured. Both are worth reading alongside this one.

To summarise: NZRT’s monitoring stack uses Prometheus for metrics, Grafana for dashboards, Loki for logs, and a set of supporting tools to make all of it work. Every production container gets a liveness probe and a readiness probe. And when something goes wrong, you’ve got four quick commands to start your investigation — pod resource usage, node resource usage, recent events, and pod describe.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.