Category Archives: 000K8S

000K8S, 01 - Core Concepts

Deployments

Welcome to the NZRT Wiki Podcast. Today we’re looking at Deployments.

So, what exactly is a Deployment in Kubernetes? Think of it as a manager for your running application. A Deployment makes sure a set number of copies of your app — called pod replicas — are up and running at all times. It also handles the tricky business of updating your app and rolling back if something goes wrong.

Let’s talk about what makes Deployments so useful. First, they work declaratively. That means you describe the state you want — say, two copies of your app running the latest image — and Kubernetes figures out how to get there. You don’t have to script every step yourself.

Second, Deployments support rolling updates. When you push a new version of your app, Kubernetes starts new pods before it stops the old ones. That way, your app stays available to users throughout the whole update process, with no downtime.

Third, if something goes wrong with an update, you can roll back. There’s a single command that reverts your Deployment to its previous state, and Kubernetes handles the rest.

And fourth, scaling. If you need more capacity, you can simply adjust the number of replicas — either directly in your configuration, or automatically using something called a Horizontal Pod Autoscaler.

Now let’s look at what a real Deployment definition looks like. The example in the wiki defines a Deployment for a WordPress application running in the production environment. Here’s what it sets up in plain terms.

It tells Kubernetes this is a Deployment object, gives it the name wordpress, and places it in the nzrt-prod namespace. It then says: keep two replicas running at all times. It uses a label — in this case, app equals wordpress — so Kubernetes knows which pods belong to this Deployment.

Inside the pod template, it defines one container, also called wordpress, running version six point five of the official WordPress image. That container listens on port 80, and it’s given an environment variable pointing it to the database — specifically, a service called mysql-service.

The definition also sets resource boundaries. Each pod requests a quarter of a CPU core and 256 megabytes of memory to start. But if it needs more headroom, it can scale up to one full CPU core and 512 megabytes of memory. This stops any single pod from overwhelming your cluster.

Finally, the update strategy is set to rolling update. The configuration allows one pod to be temporarily unavailable during a rollout, and one extra pod to come up above your normal replica count. So during an update, you might briefly have three pods running — two old, one new — before the old ones are taken down.

Now let’s go through the day-to-day commands you’ll use to manage Deployments.

To see all your Deployments in the production namespace, you run a get-deployments command scoped to that namespace. To get detailed information about the WordPress Deployment specifically, you use describe-deployment and give it the name.

Scaling is simple — there’s a scale-deployment command where you specify the new number of replicas. So if you want three copies instead of two, you set replicas to three and you’re done.

Updating the container image is done with a set-image command. You name the Deployment, name the container inside it, and provide the new image version — for example, switching from WordPress six point five to six point six.

To check on a rollout while it’s in progress, there’s a rollout-status command that gives you live feedback on how the update is going.

If something goes wrong, rollout-undo is your safety net. It reverts the Deployment to whatever it was running before, immediately.

And finally, rollout-history shows you the list of previous versions on record, so you know exactly what you can roll back to.

If you want to explore further, the wiki also points to related topics covering Pods and Containers, how Deployments relate to ReplicaSets under the hood, and how Services connect your running Deployment to the outside world.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 09 - Reference

Helm Reference

Welcome to the NZRT Wiki Podcast. Today we’re looking at Helm Reference.

So, what is Helm? Put simply, Helm is the package manager for Kubernetes. If you’ve used something like npm for Node or pip for Python, Helm plays a similar role — but for Kubernetes applications. It takes all the Kubernetes manifest files your application needs and bundles them into a single versioned, configurable unit called a chart. That makes it much easier to deploy, upgrade, and manage complex applications on your cluster.

Before we jump into commands, let’s cover four core concepts you’ll hear come up again and again. First, there’s the chart itself — that’s the package containing all your Kubernetes manifest templates. Second is a release, which is what you get when you actually run a chart in a cluster — it’s the live, running instance. Third is a repository, which is just a collection of published charts you can browse and pull from, kind of like a package registry. And fourth are values — these are the configuration overrides you supply to customise how a chart behaves when it’s deployed. Those four — chart, release, repository, values — are the building blocks of everything Helm does.

Now let’s walk through the common commands you’ll use day to day. The first thing you typically do is add a repository and update it. Think of this as registering a source so Helm knows where to find charts. Once you’ve added a repo, you can search it to find specific charts by name.

When you’re ready to deploy something, you use the install command. You give your release a name, point it at the chart you want from your repo, tell it which namespace to deploy into, and optionally tell Helm to create that namespace if it doesn’t already exist. You can also pass in a values file at this point to customise the deployment.

Once something is running, you’ll eventually want to update it — that’s where the upgrade command comes in. The syntax is very similar to install: you reference the release name, the chart, the namespace, and your values file. Helm handles rolling out the changes.

To see what’s currently deployed, you use the list command. You can scope it to a specific namespace, or ask Helm to show releases across all namespaces at once.

When you need to remove something, uninstall does the job — just give it the release name and namespace and Helm tears it all down cleanly.

One of the most useful features Helm offers is rollback. If an upgrade goes wrong, you can roll back to a previous revision — just reference the release name and the revision number you want to return to. To find out what revision numbers are available, the history command shows you a full list of past deployments for any given release.

Finally, there’s the template command. This is a dry-run tool that renders all the Kubernetes manifests Helm would generate — but doesn’t actually apply them to the cluster. It’s incredibly useful for debugging or reviewing exactly what Helm is going to do before you commit to it.

Now let’s talk about the values file, because this is where a lot of the real configuration lives. A typical values file is written in YAML and covers several key areas. You set a replica count — for example, two — to control how many instances of your application run. You define your image settings: the repository to pull from, the specific tag or version you want, and the pull policy, such as only pulling if the image isn’t already present locally. You configure your service — specifying the type, like ClusterIP for internal-only access, and the port it listens on. And you set resource constraints, which is important for cluster stability. On the requests side you declare the minimum CPU and memory your container needs to start. On the limits side you set the ceiling — the maximum it’s allowed to consume. CPU is expressed in millicores, so one hundred m is a tenth of a CPU core, and memory is in mebibytes.

Together, these values give you fine-grained control over your deployment without ever touching the chart templates themselves. You just override what you need, and Helm does the rest.

If you want to go deeper on the cluster side of things, check out the related wiki pages on kubectl and the NZRT Kubernetes Setup — both will give you context that pairs well with what you’ve just heard.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 02 - Cluster Management

Cluster Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Cluster Overview.

This episode covers the day-to-day management of the NZRT Kubernetes cluster — how you access it, how you check on its health, and how you handle common maintenance tasks. Whether you’re new to the cluster or just need a quick refresher, this should give you a solid grounding.

Let’s start with cluster access. Before you can do anything useful, you need to know which cluster you’re talking to. Kubernetes uses something called contexts to keep track of this. Think of a context as a saved profile — it knows which cluster, which user, and which namespace you want to work with.

To see all the contexts you have available, you run a command that lists them out for you. Once you know the name of the context you want, you can switch to it with a single command — you just tell kubectl, which is the main Kubernetes command-line tool, to use that context by name. If you want to double-check which context is currently active, there’s a command for that too — it simply prints the name of the one you’re on right now.

One more handy access tip: you can set a default namespace so you don’t have to keep specifying it on every command. For NZRT’s setup, the main production namespace is called nzrt-prod, and you can lock your current context to that namespace with a config command. After that, any kubectl command you run will automatically target nzrt-prod unless you say otherwise.

Now let’s talk about health checks, because knowing whether your cluster is actually healthy is one of the most important things you’ll do as an operator.

First, you’ll want to check on your nodes. Nodes are the machines that actually run your workloads — think of them as the physical or virtual servers underneath Kubernetes. A quick command gives you a summary list of all nodes and whether they’re in a ready state. If you need more detail on a specific node — things like resource capacity, conditions, and recent events — you can describe that node by name to get a full breakdown.

Next, you can check the health of the core Kubernetes components themselves, like the scheduler and the controller manager. There’s a command that queries the component statuses and gives you a simple healthy or unhealthy result for each one.

If you want a broad picture of everything running across all namespaces at once, there’s a command that pulls all resources cluster-wide — pods, services, deployments, the works. This is great for a quick sanity check.

And when something seems off, events are your best friend. Kubernetes logs events whenever something notable happens — a pod fails to schedule, a container crashes, a resource limit is hit. You can pull the events from nzrt-prod sorted by the most recent timestamp, which means the freshest and most relevant information floats to the top.

Moving on to cluster info. There are three quick commands worth knowing here. The first gives you a summary of where the key cluster endpoints are running — your API server, your DNS service. The second tells you the version of both the kubectl client and the server-side Kubernetes API, which matters a lot when you’re thinking about upgrades or compatibility. The third lists all the API resource types your cluster supports — handy when you’re trying to figure out what kinds of objects you can create or query.

Now let’s go through the common management tasks. There are five main ones you’ll reach for regularly.

First is draining a node. When you need to take a node offline for maintenance — maybe to patch the operating system or swap out hardware — you drain it first. Draining safely evicts all the pods running on that node and marks it as unschedulable, so nothing new lands on it while you’re working. You use a flag to tell Kubernetes to ignore daemonsets, which are special system pods that run on every node and don’t need to be moved.

Second is uncordoning a node. Once your maintenance is done and the node is back in service, you uncordon it. This removes the unschedulable mark and lets the scheduler start placing pods on it again.

Third is cordoning a node. This is like a softer version of draining — it marks the node unschedulable so no new pods get assigned to it, but it doesn’t touch the pods that are already running there. Useful if you want to quietly wind down a node over time.

Fourth and fifth are resource monitoring commands. You can check the live CPU and memory usage of pods in nzrt-prod, or check the same metrics at the node level across the whole cluster. These are invaluable for spotting which workloads are running hot or which nodes are under pressure before things become a problem.

That covers the essentials of your cluster day-to-day — getting in, checking the health, pulling the right info, and handling maintenance safely. If you want to go deeper, the related wiki pages on Nodes, the kubectl Reference, and the NZRT Kubernetes Architecture are great next steps.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 05 - Storage

Configmaps Secrets

Welcome to the NZRT Wiki Podcast. Today we’re looking at ConfigMaps & Secrets.

If you’ve been working with Kubernetes, you’ve probably run into the question of where to store your application’s configuration. Things like environment names, URLs, log levels, database passwords, API keys — all of that has to live somewhere. Kubernetes gives you two dedicated resources for this: ConfigMaps and Secrets. They work in a similar way, but serve different purposes, and knowing which to use and when matters a lot.

Let’s start with ConfigMaps. A ConfigMap is where you store non-sensitive configuration data. Think of it as a key-value store that your application can read at runtime. In NZRT’s setup, you might have a ConfigMap called app-config sitting in the nzrt-prod namespace. Inside it, you’d find entries like the application environment set to production, the application URL pointing to the NZRT network app endpoint, and a log level set to info. None of that is sensitive — it’s just configuration your app needs to know about.

To actually use that ConfigMap inside a pod, you reference it in your pod definition using something called envFrom. What that does is tell Kubernetes to pull all the key-value pairs from the ConfigMap and inject them into your container as environment variables. Your app then reads them just like any normal environment variable — it doesn’t need to know or care that they came from Kubernetes.

Now let’s talk about Secrets. The concept is the same — key-value pairs injected into a pod — but Secrets are designed for sensitive data. Passwords, usernames, tokens, API keys. In the example from the wiki, there’s a Secret called db-credentials, again in nzrt-prod. It holds a database password and a database username. You’ll notice the wiki uses something called stringData, which means you write the values in plain text in your definition file, and Kubernetes automatically base64-encodes them when it saves them to the cluster. Using a Secret in a pod works exactly the same way as a ConfigMap — you use envFrom, but this time you reference a secretRef instead of a configMapRef. The end result is the same: your container sees those values as environment variables.

Now here’s something really important you need to understand about Secrets, and the wiki flags this clearly. Secrets are not encrypted by default. They’re only base64-encoded when stored in etcd, which is Kubernetes’ internal data store. And base64 is not encryption — it’s just encoding. Anyone with access to etcd can decode those values trivially. So for production environments, you need to go further. There are two main approaches. First, you can enable Encryption at Rest on your cluster, which tells Kubernetes to actually encrypt the Secret data before writing it to etcd. Second, and often better for serious deployments, you use an external secrets manager. Tools like HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets are purpose-built for this and give you much stronger security guarantees, audit trails, and access controls.

Finally, let’s cover a few kubectl commands you’ll use regularly when working with these resources. To list all ConfigMaps in the nzrt-prod namespace, you run kubectl get configmaps with the namespace flag. For Secrets, same structure — kubectl get secrets with the namespace flag. If you want to inspect a specific Secret, you use kubectl describe followed by the Secret name and namespace — this shows you metadata and keys but not the actual values, which is intentional. And if you need to create a Secret quickly from the command line without a YAML file, you can use kubectl create secret generic, give it a name, pass in your key-value pairs using the from-literal flag, and specify your namespace. That’s a handy shortcut for quick setups or testing.

To link this back to broader context, ConfigMaps and Secrets connect closely to the Security Overview and Storage Overview docs in the wiki, so if you want to go deeper on cluster security or how persistent data is handled, those are your next stops.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 03 - Workloads

Deployments Replicasets

Welcome to the NZRT Wiki Podcast. Today we’re looking at Deployments & ReplicaSets.

Let’s start with the big picture. In Kubernetes, a Deployment is the object you use to manage your applications. But it doesn’t directly manage pods — it does so through something called a ReplicaSet. Think of it as a chain of command. Your Deployment sits at the top, and underneath it creates a ReplicaSet. That ReplicaSet is what actually ensures the right number of pod replicas are running at any given time.

When you update a Deployment, Kubernetes creates a new ReplicaSet to represent the new version of your application. The structure looks something like this: at the top you have your Deployment. Below that, you have your current ReplicaSet, which holds all your running pods. And separately, you also have your previous ReplicaSet — still there, not actively running pods, but kept around so that if something goes wrong, you can roll back to it.

This is an important point. You should never manage ReplicaSets directly. Always go through the Deployment. Kubernetes keeps those old ReplicaSets around for rollback history, and how many are kept is controlled by a setting called revision history limit. If you start manually changing ReplicaSets, you risk breaking that rollback chain.

Now let’s talk about how Deployments handle updates. There are two strategies available to you.

The first is called Rolling Update, and it’s the default. With a rolling update, Kubernetes gradually replaces your old pods with new ones. It doesn’t take everything down at once — it brings new pods up while slowly removing the old ones, so your application stays available throughout the process. Two parameters fine-tune this behaviour. The first is max unavailable, which sets the maximum number of pods that can be offline during the update — by default, twenty-five percent of your total pod count. The second is max surge, which controls how many extra pods can be created above your desired count during the update — also twenty-five percent by default. Together these give you a smooth, controlled rollout.

The second strategy is called Recreate. This one is much more direct. Kubernetes terminates all your existing pods first, and only then starts creating the new ones. That means there will be a period of downtime. You’d use this if your application can’t safely run two versions simultaneously — for example, if it has strict database migration requirements that would break under a mixed-version environment.

Now let’s cover scaling. You have two options: manual and automatic.

For manual scaling, you issue a command that tells Kubernetes to set the replica count for your deployment to a specific number. As a concrete example, you could scale your wordpress deployment in the nzrt-prod namespace to three replicas with a single terminal command.

For automatic scaling, you use something called a Horizontal Pod Autoscaler, or HPA. This watches a metric — typically CPU usage — and adds or removes pods automatically to keep that metric within a target range. You can set this up from the command line by specifying the deployment you want to target, a minimum and maximum replica count, and the CPU utilisation percentage that should trigger scaling. For the wordpress deployment, for example, you might say: keep at least two replicas, scale up to a maximum of five, and aim to stay at or below seventy percent CPU.

You can also define the HPA as a configuration manifest — a structured file that describes the same settings. That file would name the autoscaler, point it at your wordpress deployment, set the replica boundaries, and define the CPU metric target. Storing your HPA as a file is useful because you can check it into version control and apply it consistently across environments.

To check what your autoscaler is currently doing, you can run a command that lists all HPAs in a given namespace. That gives you a live view of current versus desired replica counts and what the resource utilisation looks like right now.

One final thing worth noting — Deployments aren’t the right tool for every workload. If you’re running stateful applications like databases, you’ll want to look at StatefulSets instead. StatefulSets give each pod a stable identity and persistent storage, which is something that ReplicaSet-backed Deployments simply don’t provide.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 04 - Networking

Ingress

Welcome to the NZRT Wiki Podcast. Today we’re looking at Ingress.

If you’ve been working with Kubernetes, you’ve probably run into the question of how external traffic actually gets into your cluster. That’s where Ingress comes in. At its core, Ingress is a Kubernetes resource that routes incoming HTTP and HTTPS traffic to the right internal services, based on things like the hostname in the request or the URL path. One important thing to know upfront is that Ingress doesn’t work on its own. You need something called an Ingress Controller running in your cluster, and at NZRT we use the nginx-ingress controller for that.

So why use Ingress at all? To understand that, it helps to compare it with the alternative, which is a LoadBalancer Service. If you compare the two side by side, you see some important differences across five areas. First, protocol support. A LoadBalancer Service works at the TCP and UDP level, meaning it can handle almost any kind of traffic. Ingress, on the other hand, is HTTP and HTTPS only. Second, SSL termination. A LoadBalancer Service doesn’t handle SSL for you, but Ingress does. Third and fourth, routing. A LoadBalancer Service has no concept of path-based or host-based routing, meaning it just forwards everything. Ingress gives you both, so you can send traffic to different services depending on the URL path or the hostname. And fifth, cost. This is a big practical one. With LoadBalancer Services you need one load balancer per service, which adds up quickly in cloud environments. With Ingress, you only need one load balancer for everything, and Ingress handles the routing internally. That’s a significant saving.

Now let’s look at what an actual Ingress configuration looks like at NZRT. The manifest defines an Ingress resource called nzrt-ingress, sitting in the nzrt-prod namespace. It includes a couple of annotations, which are basically extra instructions for the nginx controller. One tells nginx to rewrite the request path to a forward slash, and the other tells cert-manager, which is the tool we use for SSL certificates, to use our production Let’s Encrypt issuer when generating a certificate.

The spec section is where the real routing logic lives. It starts by declaring that this Ingress uses the nginx ingress class. Then it sets up TLS, specifying that the hostname app dot nzrtnetwork dot com should be secured, and that the certificate should be stored in a Kubernetes secret called nzrt-tls. Finally, there’s a rules section. It says that any request arriving at app dot nzrtnetwork dot com, regardless of the path, should be forwarded to a service called wordpress-service on port 80. So in plain terms, you hit the NZRT app URL in your browser, Ingress intercepts that, terminates the SSL, and hands the request off to the WordPress service inside the cluster.

That brings us to cert-manager, which handles the TLS certificate side of things. You install cert-manager by applying a single manifest file from the cert-manager releases page. Once it’s running, it watches for Ingress resources that reference a cluster issuer, and it automatically requests and renews certificates from Let’s Encrypt on your behalf. You don’t have to touch the certificate manually.

To check on the status of a certificate, you run a command that lists all certificates in the nzrt-prod namespace. If you want more detail on a specific one, you describe the nzrt-tls certificate in that same namespace, and Kubernetes gives you a full breakdown including whether the certificate was issued successfully, when it expires, and any events or errors that occurred during the issuance process. If something goes wrong with your SSL setup, that describe output is usually the first place you look.

To tie this all together, the flow works like this. A user visits app dot nzrtnetwork dot com. DNS resolves that to the single load balancer sitting in front of your cluster. The load balancer passes the request to your nginx Ingress controller. The controller checks its rules, sees that the hostname matches, terminates the TLS using the certificate stored in the nzrt-tls secret, and forwards the plain HTTP request to the wordpress-service inside the cluster. All of that happens transparently, and you only needed one load balancer to make it work across however many services you want to add in future.

If you want to go deeper on any of this, the related topics in the wiki cover Networking Overview, Services, and SSL and DNS, which will give you the full picture of how all these pieces connect.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 03 - Workloads

Jobs Cronjobs

Welcome to the NZRT Wiki Podcast. Today we’re looking at Jobs and CronJobs.

If you’ve worked with Kubernetes before, you’re probably familiar with Deployments — resources that keep pods running continuously. But what about tasks that just need to run once and finish? That’s where Jobs and CronJobs come in, and they’re incredibly useful for batch work like database migrations, backups, and scheduled reporting.

Let’s start with Jobs. A Job in Kubernetes runs one or more pods and keeps going until a specified number of them complete successfully. Think of it like telling Kubernetes: run this task, and don’t stop until it’s done. If a pod fails partway through, Kubernetes will try again.

The first code example shows a Job definition written in YAML. It defines a Job called “db-migration” running in the nzrt-prod namespace. Inside, it sets up a container called “migrate” using an application image, and it runs a PHP Artisan migrate command — so this is a database migration job. Two important settings here: the restart policy is set to “on failure”, meaning if the container crashes, Kubernetes will restart it rather than just giving up. And the backoff limit is set to three, which means Kubernetes will retry the job up to three times before marking it as failed.

The second code example shows three commands you’d use to work with that job from the command line. The first lists all jobs in the nzrt-prod namespace so you can see their status. The second pulls the logs from the db-migration job so you can check what actually happened during the run. And the third deletes the job once you’re done with it — important to keep things tidy, since completed jobs don’t clean themselves up automatically.

Now let’s move on to CronJobs. A CronJob is simply a Job that runs on a schedule. It wraps a Job definition inside a schedule expression, the same kind you’d use in a Linux crontab. If you’ve ever set up a scheduled task on a Linux server, this will feel very familiar.

The third code example shows a CronJob called “nzrt-backup” — again in the nzrt-prod namespace. The schedule field uses a cron expression that means “run at two in the morning, every day”. Inside, it defines a job template that runs a container using a backup tool image, executing a shell script called backup.sh. The restart policy is again set to “on failure”. There are also two history settings worth noting: successful jobs history is kept for three runs, and failed jobs history is kept for just one. This controls how many completed and failed job records Kubernetes retains, so you can look back and see what happened without cluttering the cluster with old records.

The fourth code example shows how you interact with CronJobs from the command line. The first command lists all CronJobs in the namespace. The second gives you a detailed description of the nzrt-backup CronJob — useful for checking the schedule, last run time, and any issues. The third command is particularly handy: it lets you manually trigger a CronJob right now without waiting for the schedule. You create a job “from” the CronJob definition, give it a name like “manual-backup”, and it runs immediately. This is great for testing or for running an out-of-schedule backup when you need one.

So to pull it all together — Jobs are your go-to when you need Kubernetes to run a task to completion, like a migration or a one-off data process. CronJobs wrap that same concept in a schedule so the work happens automatically and repeatedly. Both support retry logic through the restart policy and backoff limit, and CronJobs give you that manual trigger option when you need to run something on demand.

If you want to dig deeper, the related topics in the wiki cover Deployments and ReplicaSets for long-running services, and Monitoring Overview for keeping an eye on how your Jobs are performing.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 10 - Kagent

Kagent Agents Tools

Welcome to the NZRT Wiki Podcast. Today we’re looking at Kagent Agents & Tools.

This page covers the built-in tools available inside Kagent, some real example agents configured for NZRT, and how you actually talk to those agents once they’re running.

Let’s start with the built-in Kubernetes tools. These come from something called the kagent-tool-server, and there are eight of them. Think of each one as a kubectl command wrapped up so an AI agent can use it. The first lets an agent list all available resource types in the cluster — similar to running kubectl api-resources yourself. Next there’s a tool for describing a specific resource, giving you the same detail as kubectl describe. There’s one for fetching pod logs, one for listing cluster events, and one for testing service connectivity from inside the cluster — similar to running curl internally. On the write side, you have a tool for applying a YAML manifest, a tool for patching a specific field on a resource, and finally a tool for deleting resources — though the docs flag that last one as something you should leave out of most agents.

Beyond the core Kubernetes tools, there are four extended MCP servers you can bolt on. Helm gives you install, upgrade, and rollback capabilities, enabled by setting a helm flag during install. Prometheus lets you run metric queries but requires Prometheus to already be running in your cluster. Argo CD adds GitOps sync, diff, and rollback and needs Argo CD installed. And Istio gives you service mesh configuration and traffic management, which requires Istio.

Now let’s look at three example agents NZRT has defined.

The first is the SRE Agent, built for read-only diagnostics. Its configuration marks it as a Declarative agent living in the kagent namespace. The system message tells it it’s a read-only SRE agent covering the nzrt-prod, nzrt-staging, nzrt-dev, monitoring, and kagent namespaces. It’s instructed to diagnose issues, explain pod failures, recommend fixes, never apply changes, and never expose Secret values in responses. The tools it gets are all read-only — resource listing, pod logs, describe, events, and service connectivity. You’d use this agent by asking things like why a pod is crashing in production, checking service connectivity for Nextcloud, or reviewing what events happened in staging over the last hour.

The second is the Staging Deployment Agent, which has controlled write access. Its system message scopes it strictly to the nzrt-staging namespace — it can apply and patch Deployments and Services there, but is explicitly told never to touch production and never to delete resources. It also has a rule to confirm with you before applying anything. On top of the read-only tools, it gets the apply manifest and patch resource tools as well. This is the agent you’d use for rolling out a new WordPress image to staging, scaling the Nextcloud deployment to two replicas, or pushing an updated ConfigMap.

The third is the Monitoring Agent, which combines Kubernetes logs and events with Prometheus metrics. Its system message tells it to query Prometheus and pod logs, identify performance issues, and deliver plain-English summaries flagging anomalies. It pulls tools from two MCP servers — the standard kagent-tool-server for pod logs and events, and a separate Prometheus MCP server that adds two more tools: one for instant metric queries and one for querying metrics across a time range.

Finally, let’s cover how you actually reach these agents once they’re deployed. There are three options. The first is the dashboard — you run the kagent dashboard command, open your browser to localhost on port 8082, select an agent, and start chatting. The second is the command-line interface, where you run a kagent chat command and specify the agent name and namespace — for example, targeting the nzrt-sre-agent in the kagent namespace. The third is a direct HTTP API call, where you post a JSON request to a local endpoint that includes the agent name in the URL path. The message body just contains the text of your question, such as asking it to list all pods in nzrt-prod.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 10 - Kagent

Kagent Claude

Welcome to the NZRT Wiki Podcast. Today we’re looking at kagent-claude.

kagent-claude is a setup that gets kagent version 0.8.6 running on Windows 11 using Minikube, with Claude Sonnet as the AI model powering it. The session was saved back in April 2026 and at that point it was running fine, just waiting on an API credit top-up.

Let’s start with the environment. You’ve got six tools in play here. Minikube at version 1.38.1, kubectl at version 1.34.1, Helm at version 4.1.4, Docker Desktop at version 29.4.0, kagent itself at version 0.8.6, and the Claude model being used is claude-sonnet-4-6. Minikube is using Docker as its driver, running through WSL2. Memory is capped at around 3,800 megabytes, which is the Docker Desktop limit. Everything lives in a Kubernetes namespace called kagent.

Now let’s walk through the three Kubernetes resources that were created to make this work.

The first is a secret. This is how you securely pass your Anthropic API key into the cluster. The command creates a generic secret called kagent-secrets inside the kagent namespace and stores your API key under a label called ANTHROPIC_API_KEY. Think of it as a locked box that Kubernetes can open whenever it needs to authenticate with Anthropic.

The second resource is a ModelConfig, called claude-sonnet. This is essentially a configuration file that tells kagent which AI model to use and how to use it. It points to Anthropic as the provider, specifies claude-sonnet-4-6 as the model, and references that secret you just created so it can grab the API key. There is also a field for the secret key name pointing to ANTHROPIC_API_KEY, and this is important — we will come back to why in a moment. The model is configured with a maximum of 8,096 tokens and a temperature of 0.7, which gives you a balance between creative and focused responses.

The third resource is the agent itself, called claude-test-agent. This is a declarative agent using the Python runtime, pointed at your claude-sonnet model config. Its system message tells it to act as a helpful Kubernetes assistant powered by Claude Sonnet. It is also wired up to four Kubernetes tools via an MCP server: one for getting resources, one for describing a resource, one for fetching pod logs, and one for retrieving cluster events. So this agent can actually look around your Kubernetes cluster and answer questions about what is happening inside it.

Once everything is deployed, you access the UI through a port-forward command. This routes the kagent UI service from inside your cluster out to port 8080 on your local machine, so you can open a browser and go to localhost colon 8080 to start chatting with your agent.

If you ever need to resume the setup after a restart, the process has four steps. Start Docker Desktop, then run minikube start with the Docker driver, then run the port-forward command again, and finally open localhost colon 8080 in your browser.

Now, there are three known issues worth knowing about.

The first is an authentication error that says something like could not resolve authentication method. This happened because the ModelConfig was originally written with the wrong field name — the incorrect version ends in Ref, but the correct field name does not. The fix is to recreate the ModelConfig with the right field name, then bounce the agent by scaling its deployment down to zero replicas and back up to one.

The second issue is pods getting stuck in a pending state due to not enough memory. With only around 3,800 megabytes available, too many bundled agents were competing for resources. The fix was deleting the agents that were not needed — specifically the Cilium debug, manager, and policy agents, the kgateway agent, and the argo-rollouts conversion agent. Removing those freed up enough memory for everything important to run.

The third issue is a 400 error related to API credits. This is a billing distinction that is easy to miss: your Claude Pro subscription on claude.ai and your Anthropic API account at console.anthropic.com are completely separate billing systems. If you are using kagent with an API key, you need credits in the API console specifically. The fix is to top up at console.anthropic.com under settings and then billing.

So to recap: kagent-claude gives you a Claude-powered Kubernetes assistant running locally in Minikube. You set it up with a secret for your API key, a ModelConfig pointing at Claude Sonnet, and a declarative agent with four Kubernetes tools attached. The UI is a simple port-forward away. Watch out for the field name on the ModelConfig, keep memory free by pruning unused agents, and make sure your API credits are topped up separately from your Claude subscription.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 10 - Kagent

Kagent Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Kagent Overview.

So, what exactly is Kagent? At its core, Kagent is a Kubernetes-native AI agent framework. It was contributed to the Cloud Native Computing Foundation — the CNCF — by Solo.io, and it made its debut at KubeCon EU in 2025. The idea is straightforward but powerful: you define AI agents in YAML, deploy them as Kubernetes workloads, and those agents use a large language model — like Claude or one of several other options — to automate cluster operations, troubleshoot issues, and handle DevOps workflows directly inside your cluster.

What makes this interesting is that your agents aren’t some external service bolted onto your cluster. They run as standard Kubernetes Pods. That means they’re fully observable, they can be restarted, they can be scaled, and they fit right into your existing operational patterns. You get multi-LLM provider support out of the box — Anthropic, OpenAI, Azure OpenAI, Google Vertex AI, and even Ollama for local models. You also get a built-in tool ecosystem using the MCP protocol for Kubernetes operations, plus a web UI, a command-line tool called kagent, and of course the YAML-based interface you’d expect in any Kubernetes-native project.

Now let’s talk about how the architecture hangs together. Picture your Kubernetes cluster with four main moving parts working in concert.

First, you have the Controller. This is a Go deployment that sits and watches for changes to the custom resource definitions — the CRDs — that Kagent introduces. When you create or update an agent definition, the Controller is what responds and creates the underlying Pods and Services.

Second, there’s the ModelConfig. This is a CRD that defines your LLM provider — so, say, Anthropic — along with the specific model name you want to use and a reference to a Kubernetes Secret that holds your API key. This keeps your credentials managed the Kubernetes way, which is exactly what you’d want.

Third, you have the MCPServer CRD. This is what exposes the tools your agent can actually call. Think of things like kubectl operations, Helm chart management, Argo workflows, and so on. These tools follow the MCP protocol, and they’re what give your agent its ability to actually do things in the cluster rather than just talk about them.

And fourth, bringing it all together, you have the Agent CRD itself. This combines your ModelConfig, your MCPServer tool definitions, and a system prompt into a single running Pod. That’s your agent — live, running, ready to take requests.

So how does it actually work when you send a message? The flow goes like this. You send a message to the agent — through the web UI, the CLI, or the API. The agent takes your message along with its configured system prompt and sends it to Claude, or whichever LLM you’ve set up. Claude then looks at the available tools from the MCPServer and picks the right ones to use. Those tools execute against the Kubernetes API — so actions like getting pod status, reading logs, applying a manifest, or checking service connectivity happen in your actual cluster. Claude then takes those results, synthesises them, and sends back a coherent response. And throughout all of this, every action is traced via OpenTelemetry, so you have full observability into what your agent did and why.

Now, thinking about how NZRT would actually use this — there are four key patterns worth knowing about.

The first is an SRE troubleshooter agent. This is a read-only diagnostics setup, using tools to get resources, read pod logs, pull events, and describe resources. Safe, observational, great for first-line investigation.

The second is a deployment helper. This one steps it up a notch — it can generate manifests and apply them, patch resources, and check service connectivity. Your agent becomes an active participant in your deployment process.

The third is a Helm operator. Using the Helm MCP server, your agent can manage charts — installing, upgrading, rolling back — all driven by natural language instructions.

And the fourth is a monitoring analyst. Hook your agent up to the Prometheus MCP server and it can run metrics queries, surface anomalies, and give you plain-language summaries of what your cluster is doing.

If you want to go deeper from here, there are a few related wiki pages to check out. Kagent Setup covers the step-by-step install and configuration for NZRT specifically. Kagent RBAC and Security goes into service accounts, roles, and how to restrict what tools an agent is allowed to use — important for any production setup. And Kagent Agents and Tools gives you the full MCP tool reference along with example agent CRD definitions you can adapt.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 10 - Kagent

Kagent Rbac Security

Welcome to the NZRT Wiki Podcast. Today we’re looking at Kagent RBAC & Security.

This episode covers the security model for kagent running in the NZRT Kubernetes cluster. We’re talking about ServiceAccounts, role-based access control, API key management, tool restrictions, and network isolation. Let’s get into it.

The foundation of everything here is the principle of least privilege. That means each agent gets only the permissions and tools it actually needs — nothing more. At NZRT we run two agent classes. The first is the read-only class, used for SRE diagnostics. These agents can get, list, and watch any resource in the cluster, and they have access to tools like fetching resources, reading pod logs, describing resources, checking events, and testing service connectivity. The second class is the write agent, used for deployment and operations work. It has all the same read permissions, plus the ability to create and patch Deployments and Services. One thing that’s true for both classes — the destructive delete tool is never included. Full stop.

Now, each agent runs under its own dedicated ServiceAccount. The YAML definition for this is straightforward — you’re creating a ServiceAccount in the kagent namespace, giving it a name like kagent-sre, and labelling it with the NZRT owner and service code. One ServiceAccount per agent keeps permissions clean and auditable.

For the read-only SRE agent, you create a ClusterRole that grants get, list, and watch verbs across a broad set of resource types — pods, pod logs, deployments, replica sets, stateful sets, services, endpoints, events, namespaces, config maps, ingresses, jobs, and cron jobs. That ClusterRole then gets bound to the ServiceAccount via a ClusterRoleBinding. So the role defines what’s allowed, and the binding connects that role to the specific agent identity.

The write agent works a bit differently. Instead of a ClusterRole that applies everywhere, it uses a namespaced Role — scoped specifically to the staging namespace, not production. That Role allows get, list, watch, patch, and update on Deployments, and get, list, watch, create, and patch on ConfigMaps and Services. If you ever need production write access, the wiki is clear: that requires an explicit review before it gets granted. No shortcuts there.

Let’s talk API key security, because this one matters a lot. There are four rules NZRT follows. First, API keys never go in a ConfigMap — they live in a Kubernetes Secret only. Second, they never go in Git — you create them directly using the kubectl command or an external secrets operator. Third, the Anthropic key is namespace-scoped, meaning it lives only in the kagent namespace. And fourth, rotation is handled by updating the secret value, and any pod that restarts will automatically pick up the new key. The rotation command shown in the wiki takes the new key value and applies it using a dry-run pipeline to update the existing secret in place — clean and non-destructive.

Tool restriction is next. Each agent has an explicit whitelist of tool names. If a tool name isn’t on the list, the agent simply can’t use it. For a read-only agent, the whitelist includes things like listing API resources, reading pod logs, describing resources, and fetching cluster events. Tools like applying manifests are commented out with a note that they’re not included for read-only agents. And the delete resource tool? Also commented out, with a note that it’s never included. The guidance here is to start with the minimum set and only add tools when you have a specific, justified use case.

Network isolation is the last infrastructure control. A NetworkPolicy restricts what the kagent pod can reach on egress. It allows outbound TCP on port 443 to the Kubernetes API server — using the specific IP range for your cluster — and outbound TCP on port 443 to reach the Anthropic API. That’s it. The pod can’t freely reach the internet or other internal services.

Finally, every agent’s system prompt should include a set of safety instructions. These are written in plain language: always use informational tools before any modification tools, never delete resources, never expose Secret values in responses, and for any destructive or risky action, output the command but wait for human confirmation before proceeding. These rules act as a last line of defence at the model level.

Put it all together and you’ve got defence in depth — ServiceAccounts isolate identity, RBAC limits what each identity can do, Secrets protect credentials, tool whitelists constrain agent behaviour, network policies limit egress, and system prompt rules guide the model itself.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 10 - Kagent

Kagent Setup Nzrt

Welcome to the NZRT Wiki Podcast. Today we’re looking at Kagent Setup — NZRT.

This episode walks you through installing kagent in the NZRT Kubernetes cluster and wiring it up to use Claude Sonnet as the primary language model. By the end you’ll have a working SRE agent running inside the cluster and ready to diagnose problems.

Let’s start with what you need before you begin. There are five prerequisites. First, a Kubernetes cluster — NZRT already has prod, staging, and dev namespaces set up. Second, kubectl configured with cluster access. Third, Helm version 3 for the chart installation. Fourth, an Anthropic API key from the NZRT Gmail Anthropic account. And fifth, the kagent CLI itself, which we install in the very first step.

Step one is getting the kagent CLI onto your machine. You run a one-line install script that downloads and runs the installer automatically from the kagent website. Once that finishes, you check that it worked by asking for the version number. If a version comes back, you’re good to move on.

Step two is creating the kagent namespace in your cluster. A single kubectl create namespace command handles that. You then label the namespace with three NZRT standard values — owner set to nzrt, service-code set to 000K8S, and environment set to prod. Those labels keep things consistent with how NZRT organises all its cluster resources.

Step three is storing your Anthropic API key as a Kubernetes secret. You create a secret called anthropic-key inside the kagent namespace, passing your API key in as a literal value. You then run a get secret command to verify it’s there. Once you see it listed, the key is safely stored and ready to use.

Step four is the main Helm installation. You add the kagent chart repository, update your local cache, then run the install command. That command puts kagent into the kagent namespace and you pass in three settings: the default provider is Anthropic, the secret name is the one you just created, and you specify which field inside that secret holds the actual key value. After the install, you check the rollout status of the kagent controller deployment and list all pods in the namespace to confirm everything is running.

Step five is creating a ModelConfig resource. This is how you tell kagent exactly which Claude model to use. You write a small YAML file defining a resource of kind ModelConfig, name it claude-sonnet, place it in the kagent namespace, and add the NZRT owner and service-code labels. In the configuration spec you reference the anthropic-key secret, set the model to claude-sonnet-4-6, and set the provider to anthropic. You apply the file with kubectl and the model config becomes live. The wiki also notes that for heavier reasoning tasks you can create a second ModelConfig pointing at claude-opus-4-7.

Step six is deploying the NZRT SRE agent, and this is where everything comes together. The agent is defined in another YAML file, this time as a resource of kind Agent. It’s named nzrt-sre-agent, lives in the kagent namespace, and carries the standard NZRT labels. The most important part is its system message — the instruction set that defines what the agent does and how it behaves. It tells the agent it is an SRE agent for the NZRT cluster, lists the namespaces it works across — prod, staging, dev, monitoring, and kagent — and sets firm boundaries. It must use read-only tools first, it must never delete or modify resources, and when it has a fix recommendation it should output the exact command or YAML but wait for human confirmation before anything gets applied. The agent is linked to the claude-sonnet ModelConfig from the previous step. It also gets five tools from the kagent tool server: get resources, get pod logs, describe a resource, get cluster events, and check service connectivity. You apply the file with kubectl just as before.

Step seven is opening the dashboard. A single kagent dashboard command launches a local web interface at port 8082. If you prefer to do it manually, you can use kubectl to port-forward the kagent dashboard service to that same port on your machine.

To know that everything is working correctly, run through this mental checklist. All pods in the kagent namespace should show as Running. Your claude-sonnet ModelConfig and your nzrt-sre-agent should both show a Ready status. The dashboard should be reachable at localhost on port 8082. And when you send the agent a test message asking it to list all pods in nzrt-prod, it should respond with that pod list — confirming the model connection and tool access are both functioning end to end.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 09 - Reference

Kubectl Cheatsheet

Welcome to the NZRT Wiki Podcast. Today we’re looking at kubectl Cheatsheet.

This episode is your quick-reference guide for daily kubectl usage in the NZRT cluster. Whether you’re just getting started or need a fast reminder of the commands you reach for most often, we’re going to walk through everything section by section.

Let’s start with context and namespace. Before you do anything else in kubectl, you need to make sure you’re talking to the right cluster and working in the right namespace. There’s a command that lists all the contexts you have configured, so you can see what’s available. Then there’s a command to switch to a specific context by name. Finally, you can pin your current context to a specific namespace — in NZRT’s case, that’s the nzrt-prod namespace — so you don’t have to type it out every single time. Getting these three things right at the start saves you a lot of headaches later.

Next up is getting resources. Once you’re in the right context, you’ll want to see what’s running. You can pull a combined view of your pods, services, deployments, and ingresses all in one go, scoped to nzrt-prod. If you want everything in that namespace, there’s a command that literally gets all resources at once. You can also go wider and get everything across all namespaces in the cluster. And if you want to see your nodes with extra detail like IP addresses and which zone they’re in, there’s a wide-output option for that too.

Now let’s talk about inspecting things more deeply. This is where you go when something looks off. You can describe a specific pod by name, which gives you a detailed breakdown of its current state, events, and configuration — really useful for diagnosing problems. You can also stream the live logs from a pod, which is handy when you’re watching something in real time. If you need to actually get inside a running container and poke around, there’s an interactive shell command that drops you straight into a shell session inside the pod. And if you need to hit a service locally without exposing it externally, port forwarding lets you map a local port on your machine to a port on the service.

Moving on to applying and deleting resources. The bread and butter of day-to-day Kubernetes work. You can apply a manifest file to create or update resources defined in it. The same file can be used to delete those resources cleanly. And if you need to force-delete a pod immediately without waiting for a graceful shutdown, there’s a variant of the delete command that skips the grace period entirely — useful when a pod is stuck terminating.

Now for scaling and updating deployments. If you need to scale a deployment up or down, you specify the deployment name and the number of replicas you want, and Kubernetes handles the rest. Updating the container image inside a deployment is just as straightforward — you name the deployment, the container, and the new image with its tag. After triggering a rollout, you can watch its progress with the rollout status command, which tells you when it’s done or if something went wrong. And if the new version causes problems, there’s a rollout undo command that takes you straight back to the previous version with a single command. That one’s a lifesaver.

Let’s cover debugging next, because things do go wrong. The events command gives you a chronological list of what’s been happening in the namespace, sorted by the most recent timestamp — great for piecing together what just broke and when. You can also check resource consumption with the top commands, which show you CPU and memory usage for pods and nodes respectively. If you’re unsure whether your current permissions allow a certain action, there’s a command to check whether you’re allowed to do something like create pods in a namespace. And finally, you can spin up a temporary debug pod running a minimal image, drop into a shell, do your troubleshooting, and have it automatically cleaned up when you exit. That one’s great for checking network connectivity or DNS from inside the cluster.

The last section is about output formatting. Sometimes you need the raw data in a specific format. You can get the full YAML definition of any pod, which is useful when you want to see exactly how it’s configured or copy it as a base for a new manifest. You can also extract a specific field — like just the pod’s IP address — using a path expression that navigates the JSON structure. If you want to see which labels are attached to your pods, there’s a flag that adds a labels column to the output. And you can filter pods by label too, so for example you could list only the pods belonging to the WordPress app.

That covers the full kubectl cheatsheet for the NZRT cluster. Context first, then inspect, apply, scale, debug, and format your output. Keep these patterns in mind and you’ll move through cluster work quickly and confidently.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 02 - Cluster Management

Kubectl Reference

Welcome to the NZRT Wiki Podcast. Today we’re looking at kubectl Reference.

If you’re working with the NZRT Kubernetes cluster, kubectl is your main command-line tool. It’s how you talk to the cluster — listing what’s running, checking logs, deploying changes, and fixing problems. This episode walks you through the key commands grouped by what you’re trying to do.

Let’s start with getting and listing resources. This is probably the first thing you do when you sit down to check on the cluster. You can list pods, deployments, services, ingresses, config maps, secrets, persistent volume claims, and nodes — all within the nzrt-prod namespace. There’s also a shortcut to get everything in that namespace in one go, or if you need a full picture, you can pull everything across all namespaces at once. Think of these commands as your dashboard — a quick way to see what’s alive and what’s there.

Once you spot something you want to know more about, you move into the describe commands. Describing a resource gives you the detailed view — events, conditions, configuration specifics. You can describe a pod by name, a deployment, a node, or a service. This is especially useful when something isn’t behaving as expected and you need to dig past the surface-level status.

Next up is logs. When something goes wrong, logs are usually your first port of call. The basic version pulls logs from a named pod in nzrt-prod. If your pod is running multiple containers, you can specify which container you want logs from. If you want to watch logs live as they come in, there’s a follow mode — it streams new output to your terminal in real time, similar to tailing a file. And if a pod has already crashed, there’s a flag to pull the logs from the previous run, which is invaluable for diagnosing what went wrong before a restart.

Now for applying and deleting resources. When you have a manifest file — that’s a YAML file describing your Kubernetes resources — you can apply it to the cluster. Kubernetes will create or update whatever the file describes. You can also point at an entire directory of manifest files and apply them all at once. On the flip side, you can delete resources using the same manifest file, or delete a specific pod or deployment by name directly. Be careful with deletes — there’s no confirmation prompt, so make sure you know what you’re targeting.

The exec and port-forward commands are your tools for getting hands-on with a running pod. Exec lets you open an interactive shell session directly inside a container — either a basic shell or bash, depending on what the container has available. This is handy for inspecting files, running quick diagnostics, or testing connectivity from inside the cluster. Port-forward is different — it creates a tunnel from your local machine into the cluster. So if you specify local port 8080 mapping to port 80 on a pod or service, you can open your browser and hit that service directly from your laptop without exposing it externally. Really useful for testing.

Rollouts are how you manage deployments over time. You can check the status of a rollout to see if a deployment is progressing or stuck. You can view the rollout history to see previous versions. If something goes wrong after a deploy, you can undo it — this rolls the deployment back to its previous state. And if you want to update which container image a deployment is running, you can set that directly by specifying the deployment name, the container name, and the new image and tag.

Finally, output formats. By default kubectl gives you a summarised table view, but sometimes you need more. You can output a pod’s full configuration as YAML or JSON — useful if you want to inspect every field or save a copy. The wide output format adds extra columns to the default table, like which node a pod is running on. And you can show labels on your pods, which is helpful when you’re working with selectors or trying to understand how services are routing traffic.

That covers the full kubectl reference for NZRT cluster management. If you want to go deeper, the related wiki pages on the kubectl Cheatsheet and Cluster Overview are good next reads.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

00 - Overview, 000K8S

Kubernetes Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Kubernetes Overview.

So, what exactly is Kubernetes? You might have seen it written as K8s, which is just a shorthand where the eight stands for the eight letters between the K and the s. At its core, Kubernetes is an open-source platform for container orchestration. That’s a fancy way of saying it automates the deployment, scaling, and management of containerised applications. Rather than you manually keeping track of which containers are running where, Kubernetes handles all of that on your behalf.

The way it works is by grouping containers into units called Pods, and then managing those pods across a cluster of machines called Nodes. Let’s talk about what Kubernetes actually does for you. First, it handles scheduling, which means it decides which pod gets placed on which node based on available resources. Second, it takes care of self-healing. If a container crashes or a node stops responding, Kubernetes automatically restarts or replaces it without you having to intervene. Third, it supports scaling. If your application suddenly needs to handle more traffic, Kubernetes can automatically spin up more instances based on things like CPU or memory usage. Fourth, it supports rolling updates, meaning you can push a new version of your application with zero downtime. Fifth, it provides service discovery, using internal DNS and load balancing so your services can find each other. And sixth, it manages secrets, storing sensitive information like passwords and API keys in encrypted storage.

Now let’s talk about how Kubernetes is actually structured. The architecture has two main sides: the Control Plane and the Worker Nodes.

The Control Plane is the brain of the operation. It contains four key components. The first is the API server, which is the central REST endpoint that every Kubernetes operation goes through. Think of it as the front door to the entire cluster. The second is etcd, which is a distributed key-value store that holds the entire state of your cluster. If Kubernetes needs to know what’s running and where, it looks here. The third is the scheduler, which is responsible for deciding where new pods should run based on what resources are available across your nodes. The fourth is the controller manager, which runs a collection of controllers that handle things like deployments, node management, and network endpoints, making sure the actual state of the cluster matches what you’ve asked for.

On the other side, you have the Worker Nodes. These are the machines where your actual application containers run. Each worker node has three components. The kubelet is an agent that sits on the node and makes sure the containers assigned to it are actually running properly. The kube-proxy manages the network rules on each node so that traffic gets routed correctly to the right services. And finally there’s the container runtime, which is the software that actually runs your containers, things like containerd or Docker.

So when you ask Kubernetes to run your application, your request goes through the API server, the scheduler figures out the best node for it, the controller manager keeps an eye on it, and the kubelet on the chosen node makes sure the container stays up and running.

Now, Kubernetes uses a set of objects to represent everything in your cluster. There are eight key ones worth knowing. A Pod is the smallest deployable unit and can contain one or more containers that share the same network and storage. A Deployment manages groups of pods and handles things like rolling updates and maintaining a certain number of running replicas. A Service gives you a stable network endpoint for a set of pods, so even if individual pods come and go, traffic can still reach your application reliably. An Ingress lets you define HTTP and HTTPS routing rules for traffic coming into the cluster from outside. A ConfigMap lets you store non-sensitive configuration data separately from your application code. A Secret is similar but designed for sensitive data like passwords, tokens, and keys. A Namespace gives you a way to create virtual clusters within your physical cluster, which is useful for separating teams, environments, or projects. And finally, a PersistentVolume represents a storage resource at the cluster level, so your data can survive beyond the life of any individual pod.

Together these objects give you a powerful, declarative way to describe exactly how your applications should run, and Kubernetes takes care of making that reality happen and keeping it that way.

If you want to go deeper, the NZRT wiki has a page on the specific cluster design used here at NZRT, a page on core concepts covering pods, deployments, services, and namespaces in more detail, and a kubectl cheatsheet with the most common commands you’ll reach for day to day.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 07 - Monitoring

Logging

Welcome to the NZRT Wiki Podcast. Today we’re looking at Logging.

Let’s start with the big picture. In Kubernetes, when your containers run, they write output to two standard streams — standard output and standard error. That output doesn’t just disappear. The kubelet, which is the agent running on each node, captures it and writes it to the node’s file system. From there, a log shipper picks it up and sends it on to a centralised log storage system called Loki. And then Grafana sits on top of Loki so you can actually query and visualise everything. That’s the full chain — container output, to kubelet, to log shipper, to Loki, to Grafana.

Now let’s talk about how you access logs directly using kubectl. This is your first line of investigation when something looks wrong.

The most basic command gets the current logs from a pod. You specify the pod name and tell it which namespace to look in — in NZRT’s case that’s nzrt-prod. This gives you a snapshot of what that pod has written to its output streams up to this point.

If you want to watch logs as they come in live, you can follow them. This keeps the stream open and prints new lines as they arrive — useful when you’re actively watching a deployment or waiting for a specific event to show up.

Now, what if a pod has crashed and restarted? That’s where the previous flag comes in. When a container crashes and Kubernetes restarts it, the logs from the crashed container are gone from the current session — but you can still retrieve them by asking for the previous instance. This is often the most useful thing to look at when you’re debugging a crash.

If you’re working with a pod that runs more than one container inside it, you need to also specify which container you want logs from. You give both the pod name and the container name, and kubectl knows exactly where to look.

And finally, if you don’t know the exact pod name but you know the application label, you can ask for logs from all pods matching that label at once. For example, you could ask for all pods labelled as WordPress in nzrt-prod, and you’ll get output from every one of them in one go.

Now let’s go deeper into the log architecture — the pipeline that makes centralised logging work.

First, your pod writes to standard output or standard error. The kubelet on that node picks it up and writes it to a path on the node’s local file system, under a directory called var log pods.

From there, a DaemonSet runs on every node in the cluster. A DaemonSet means one copy of a pod runs on every single node automatically. At NZRT, that’s either Promtail or Grafana Alloy — both are log shippers that read the files the kubelet wrote and forward them on.

The destination is Loki, a log storage system designed specifically for Kubernetes environments. Loki stores your logs and indexes them by labels like namespace and app name, rather than indexing the full text of every log line. This keeps storage lean and efficient.

Finally, Grafana sits in front of Loki. This is where you go to actually query your logs, build dashboards, or set up alerts based on log content.

So the flow is: pod, to kubelet, to Promtail or Alloy, to Loki, to Grafana. Each step has a clear role.

Once you’re in Grafana, you query Loki using a language called LogQL. Let me walk you through a few examples of what those queries do in plain terms.

The most basic query selects all logs from a specific namespace and application — for example, show me everything from the nzrt-prod namespace where the app is WordPress.

You can filter further by looking for specific text in the log lines. For example, you might ask for all logs in the nzrt-prod namespace that contain the word error somewhere in the line. That’s a simple text filter and a quick way to spot problems.

If your logs are structured as JSON, which is common with modern applications, you can parse them and then filter on specific fields. So you could ask for logs where the parsed level field equals error — which is more precise than just searching for the word error anywhere in the raw text.

And if you want to understand log volume rather than read individual lines, you can use a rate query. This tells you how many log lines are being produced per second or per minute, averaged over a time window like five minutes. This is useful for spotting sudden spikes in log output, which often signals something going wrong upstream.

That’s the core of Kubernetes logging at NZRT — kubectl for direct pod access, a DaemonSet-based pipeline to ship logs into Loki, and Grafana with LogQL for querying and visualising everything. If you want to go further, check out the Monitoring Overview and the kubectl Reference pages in the wiki.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 07 - Monitoring

Monitoring Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Monitoring Overview.

If you’re running workloads on NZRT’s Kubernetes cluster, you need to know how the observability stack is put together. Observability is really just a fancy word for being able to see what’s happening inside your systems — measuring performance, viewing logs, and checking whether your services are healthy. Today we’ll walk through the tools involved, how health probes work, and some quick commands you can use to check on things at any time.

Let’s start with the tools. NZRT uses six components to cover observability end to end. First, there’s Prometheus, which handles metrics scraping and storage — think of it as the thing that constantly collects numbers from your cluster, like CPU usage, memory, and request counts. Second is Grafana, which sits on top of Prometheus and turns all those numbers into dashboards and alerts you can actually read. Third is Loki, which handles log aggregation — it gathers log output from across your cluster so you can search and query it in one place. Fourth is Promtail, also known as Alloy, which is the log shipper that runs at the node level and feeds data into Loki. Fifth is kube-state-metrics, which exposes cluster state as metrics — things like whether a deployment has the right number of replicas, or whether a pod is pending. And sixth is metrics-server, which powers the kubectl top command so you can see live resource usage on pods and nodes.

Now, five of those six tools — Prometheus, Grafana, Loki, Promtail, and kube-state-metrics — all live in a namespace called monitoring. The last one, metrics-server, lives in kube-system, which is the core Kubernetes namespace reserved for system components.

Next up: health probes. These are a critical part of running production containers, and NZRT requires you to add them to every production container you deploy. There are two types, and they serve different purposes.

The first is a liveness probe. This is Kubernetes asking the question: is this container still alive and worth keeping around? In the example configuration, the liveness probe makes an HTTP GET request to the slash health path on port 80. It waits 30 seconds after the container starts before making its first check, then checks every 10 seconds after that. If the container fails to respond three times in a row, Kubernetes will restart it.

The second is a readiness probe. This one asks a different question: is this container ready to receive traffic? It’s similar in setup — it checks a path called slash ready on port 80 — but it starts checking sooner, after just 10 seconds, and checks more frequently, every 5 seconds. The failure threshold is still three. The key difference is what happens on failure: a failed readiness probe doesn’t restart the container, it just removes it from the load balancer rotation until it recovers. That’s an important distinction. Liveness failures restart. Readiness failures pause traffic.

Together, these two probes give Kubernetes the information it needs to manage your containers intelligently without any manual intervention from you.

Now let’s talk about quick checks — commands you can run at any time to get a fast read on what’s happening in the cluster.

The first command shows you resource usage across all pods in the production namespace. You’ll see each pod’s CPU and memory consumption at a glance. The second command is similar but zooms out to the node level, showing you how much each underlying machine is using overall.

The third command pulls recent events from the production namespace and sorts them by timestamp, so the most recent things that happened in the cluster are at the bottom. This is a great first stop when something seems off — events will often tell you about failed pulls, scheduling issues, or restarts before you even go looking.

The fourth command lets you describe a specific pod by name. When you run it, pay particular attention to two sections in the output: Conditions and Events. Conditions tell you the current state of the pod — whether it’s initialized, ready, and scheduled. Events tell you the history of what’s happened to it, including any probe failures, restarts, or errors. Between those two sections, you can usually diagnose most common pod issues without needing to dig any further.

If you want to go deeper from here, the wiki also covers Logging in its own article, and there’s a Cluster Overview page that gives you the broader picture of how the NZRT Kubernetes environment is structured. Both are worth reading alongside this one.

To summarise: NZRT’s monitoring stack uses Prometheus for metrics, Grafana for dashboards, Loki for logs, and a set of supporting tools to make all of it work. Every production container gets a liveness probe and a readiness probe. And when something goes wrong, you’ve got four quick commands to start your investigation — pod resource usage, node resource usage, recent events, and pod describe.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 02 - Cluster Management

Nodes

Welcome to the NZRT Wiki Podcast. Today we’re looking at Nodes.

So, what is a node? In a Kubernetes cluster, a node is essentially a worker machine. Think of the cluster as a factory, and nodes are the individual workstations where the actual work gets done. Every node runs three key pieces of software to make that happen.

The first is the kubelet. This is an agent that sits on the node and keeps in constant communication with the Kubernetes API server. Its main job is managing the lifecycle of pods — making sure the right containers are running, healthy, and doing what they’re supposed to be doing.

The second component is kube-proxy. This one handles networking. It maintains the rules that allow network traffic to flow correctly to your services — so when something inside the cluster needs to talk to something else, kube-proxy makes sure that traffic lands in the right place.

The third component is the container runtime. For most setups you’ll see at NZRT, that’s containerd. This is what actually pulls and runs your containers at the low level.

Now, how do you check on your nodes? There are a few commands worth knowing. The first lets you list all nodes in your cluster — you get a quick overview of their names and current status. If you want more detail, you can run a wider version of that command, which also shows you IP addresses and operating system information. And if you really need to dig into a specific node — say you’re troubleshooting or checking recent events — you can describe that node by name and get a full breakdown of everything happening on it.

When you look at node status, you’ll see what are called conditions. There are five to be aware of. Ready is the one you want to see — it means the node is healthy and able to accept new pods. If you see MemoryPressure, the node is running low on memory. DiskPressure means disk space is getting tight. PIDPressure tells you there are too many processes running on that node. And NetworkUnavailable means the network isn’t configured correctly on that node. Any condition other than Ready being active is worth investigating.

The last area covered in this doc is labels and taints — two ways you can control how pods get scheduled onto nodes.

Labels are straightforward. You can tag a node with a key-value pair — for example, marking a node as belonging to a production environment. Once a node has a label, you can use that label in your pod configurations to tell Kubernetes where you want things to run.

Taints are a bit more powerful. When you taint a node, you’re essentially putting up a sign that says “pods can’t schedule here unless they explicitly say they’re okay with this.” You set a taint with a key, a value, and an effect. The effect used in this example is NoSchedule, which means Kubernetes will not place any new pods onto that node unless those pods have a matching toleration defined. This is useful for reserving nodes for specific workloads — for example, keeping a node dedicated to production traffic.

And when you’re done with a taint and want to remove it, you run the same command but add a minus sign at the end to strip it off.

So to recap — nodes are your worker machines, made up of the kubelet, kube-proxy, and a container runtime. You can inspect them with a handful of commands, watch their conditions to catch health issues early, and use labels and taints to control exactly what runs where.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

00 - Overview, 000K8S

Nzrt K8S Architecture

Welcome to the NZRT Wiki Podcast. Today we’re looking at NZRT K8s Architecture.

So what is this page all about? It covers NZRT’s specific approach to running Kubernetes — that’s the container orchestration platform you might have heard of — including how the cluster is designed, how workloads are organised, and how deployments are automated.

Let’s start with the cluster overview. The platform itself is still being decided, and the version and node count will be confirmed once the cluster is actually provisioned. What is already locked in is the container runtime — NZRT will be using containerd, which is one of the most widely adopted and production-proven runtimes in the Kubernetes ecosystem.

Next, let’s talk about the namespace strategy. If you’re new to Kubernetes, namespaces are logical partitions inside the cluster — a way to separate different environments and concerns so they don’t interfere with each other. NZRT has five namespaces planned. The first is nzrt-prod, which is where all production workloads live. Then there’s nzrt-staging, the pre-production environment — also called UAT, or User Acceptance Testing — where things get validated before going live. After that you have nzrt-dev, the space for development and active testing work. Then there’s a namespace called monitoring, which hosts the observability stack — specifically Prometheus for metrics collection and Grafana for dashboards and visualisation, along with the logging stack. And finally there’s ingress-nginx, which runs the ingress controller — that’s the component responsible for routing external web traffic into the right services inside the cluster.

Now let’s look at workload placement — which specific services run where, and how they’re configured. There are four services mapped out so far. WordPress runs in the production namespace as a Deployment, and it connects to an external MySQL database rather than one running inside the cluster itself. Dolibarr, which is NZRT’s ERP and CRM platform, also runs in the production namespace as a Deployment. Nextcloud, the file and collaboration platform, runs in production as a StatefulSet. If you’re wondering what the difference is, StatefulSets are used when a workload needs stable storage that persists across restarts — which Nextcloud definitely requires. And finally, there’s a planned blockchain node for future use — also a StatefulSet in production — which will connect to the Base network once it’s brought online.

Now let’s cover the CI/CD integration. That stands for Continuous Integration and Continuous Deployment, and it’s the automated pipeline that takes your code changes and gets them running in the cluster without manual steps in between. The diagram in the documentation shows a flow that starts at GitHub, where all code repositories live under the NZRT source control setup. From there, a GitHub Actions workflow kicks in automatically. That workflow does two things: first it builds a Docker container image and pushes it to a container registry, and then it applies the updated configuration to the Kubernetes cluster — either directly or through a tool called Helm, which is a package manager for Kubernetes that simplifies managing complex deployments. The end result is a rolling deployment, meaning the update rolls out gradually so there’s zero downtime — your users stay connected while the new version comes up in the background.

The documentation also points to a few related resources if you want to go deeper. There’s a general Kubernetes Overview page, the Infrastructure Vault under the code 000INF, and the GitHub Vault under 000GIT.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 08 - NZRT Config

Nzrt Kubernetes Setup

Welcome to the NZRT Wiki Podcast. Today we’re looking at NZRT Kubernetes Setup.

This wiki covers how NZRT configures and manages its Kubernetes cluster — from the initial namespace setup right through to workload deployment conventions. Let’s walk through it together.

First, a quick look at the cluster details. The provider, Kubernetes version, and node count are all still to be confirmed — they’ll be updated once the cluster is provisioned. What we do know is that if a cloud provider is used, the cluster will be hosted in New Zealand. And when it comes to access, the kubeconfig file — which is the credential file that lets you connect to the cluster — is stored securely in the Nextcloud vault under the 000NCL section.

Next up is namespace setup. In Kubernetes, namespaces let you divide a single cluster into logical sections, keeping different environments and concerns separated. NZRT sets up five namespaces when bringing a cluster online. You create one called nzrt-prod for production workloads, one called nzrt-staging for staging, and one called nzrt-dev for development. Then you add a monitoring namespace for observability tools, and finally an ingress-nginx namespace for the ingress controller. These five namespaces form the foundation of every NZRT cluster.

Now let’s talk about standard labels. Every resource you deploy in the NZRT cluster should carry a consistent set of labels. These labels are key-value pairs attached to Kubernetes objects that help you identify, filter, and manage resources at scale. The label set NZRT uses has five fields. First is app, which you set to the name of your application. Second is environment, which should be prod, staging, or dev depending on where the workload lives. Third is owner, which is always set to nzrt. Fourth is service-code, which carries the value 000K8S. And fifth is managed-by, which tells you whether the resource was deployed using Helm or kubectl directly. Applying these labels consistently across everything you deploy makes cluster management much easier down the line.

Moving on to ingress setup. NZRT uses nginx as its ingress controller — this is the component that routes external HTTP and HTTPS traffic into your cluster services. To install it, you first add the ingress-nginx Helm repository, then use Helm to install the chart into the ingress-nginx namespace. That one Helm command handles the full installation, and if the namespace doesn’t already exist, it creates it automatically.

After ingress, you’ll want to set up cert-manager. This is what handles automatic TLS certificate provisioning for your services — so your apps get HTTPS without you having to manually manage certificates. You install cert-manager by applying a single manifest directly from the cert-manager GitHub releases page. Once that’s done, you configure a ClusterIssuer resource. Think of a ClusterIssuer as a cluster-wide object that tells cert-manager where and how to request certificates. NZRT’s ClusterIssuer is named letsencrypt-prod and it points to the Let’s Encrypt production certificate authority. It uses the ACME protocol with an HTTP challenge — meaning Let’s Encrypt verifies domain ownership by making an HTTP request through your nginx ingress. The contact email registered with Let’s Encrypt for NZRT is nzrtnetwork at gmail dot com, and the private key generated during that registration is stored in a Kubernetes secret also named letsencrypt-prod.

Finally, let’s run through the NZRT workload checklist. Before you consider any workload ready for deployment, there are eight things you need to confirm. One — your namespace has been created and a ResourceQuota is in place to prevent any single namespace from consuming too many cluster resources. Two — RBAC roles have been applied, meaning the right permissions are granted to the right service accounts. Three — any secrets your app needs have been created directly in the cluster and are not committed to a Git repository. Four — liveness and readiness probes are configured so Kubernetes knows when your app is healthy and when it is ready to serve traffic. Five — resource requests and limits are set on your containers so the scheduler can place them correctly and prevent resource starvation. Six — your ingress is configured with TLS so traffic to your service is encrypted. Seven — if your workload needs persistent storage, that has been provisioned, typically using StatefulSets. And eight — your CI/CD pipeline in GitHub Actions is configured and connected to the cluster.

If you want to go deeper, the related pages to check out are the NZRT K8s Architecture doc and the GitHub vault under 000GIT for CI/CD pipeline details.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 05 - Storage

Persistent Volumes

Welcome to the NZRT Wiki Podcast. Today we’re looking at Persistent Volumes.

If you’ve worked with Kubernetes at all, you’ve probably run into the question of where your data actually lives. Containers are ephemeral by design — when one stops, its local data disappears with it. Persistent Volumes solve that problem, and that’s exactly what we’re covering today.

Let’s start with the two core concepts. A PersistentVolume, or PV, is a storage resource that exists at the cluster level. Think of it as a chunk of storage that Kubernetes knows about and can hand out. A PersistentVolumeClaim, or PVC, is how a workload actually requests that storage. Claims are scoped to a namespace, meaning they live within a specific environment in your cluster, and when you create one, Kubernetes binds it to an available PV that matches what you asked for.

Before you can claim storage, you usually need a StorageClass. This is where dynamic provisioning comes in. The first example in the wiki shows a StorageClass definition. It sets a name — in NZRT’s case that’s called nzrt-standard — and points to a provisioner, which is the component that actually goes and creates the underlying storage. The example uses a placeholder provisioner that you would swap out for your real cloud provider’s provisioner in a live deployment. Two other settings matter here: the reclaim policy is set to Retain, which we’ll come back to shortly, and the volume binding mode is set to wait for the first consumer, meaning storage isn’t actually provisioned until something tries to use it.

Next up is the PersistentVolumeClaim itself. The example shows a claim named nextcloud-pvc, sitting in the nzrt-prod namespace. It requests twenty gigabytes of storage, references the nzrt-standard StorageClass you just defined, and specifies an access mode of ReadWriteOnce. That access mode means only one node in the cluster can mount this volume for reading and writing at a time. That’s fine for something like a Nextcloud instance where you have one primary pod handling file access.

Once you have your claim, you need to attach it to a pod. The third example shows how that works inside a pod specification. You define a volume entry that references your claim by name — nextcloud-pvc in this case — and then inside your container definition, you mount that volume at a specific directory path. Here it’s mounted at the path where Nextcloud expects its web files to live. From the container’s point of view, it just sees a regular folder. The fact that it’s backed by persistent storage is completely transparent to the application.

Now let’s talk about the commands you’ll use day to day. You can list all persistent volumes across the cluster, list all claims within the nzrt-prod namespace specifically, pull up a detailed description of a particular claim to check its status or spot any error messages, and list all available StorageClasses. Those four commands cover most of what you need for routine inspection and troubleshooting.

Finally, reclaim policies — and this one is worth paying close attention to before you start deleting things. The wiki covers three options. Retain means that when you delete a PVC, the underlying PV stays intact and its data is preserved, but you are responsible for cleaning it up manually. Delete means that when the claim is removed, Kubernetes automatically removes the PV and all the backing storage with it. And Recycle is a third option that exists mostly as a historical footnote — it is deprecated, so don’t use it.

For most production workloads at NZRT, Retain is the safer default because it protects you from accidental data loss. Delete makes sense only when you are confident that automatically removing the data is acceptable, such as with temporary or test environments.

If you want to go deeper, the related topics in the wiki are Storage Overview and StatefulSets. StatefulSets in particular pair closely with Persistent Volumes when you need stable, predictable storage for applications that need to remember their state across restarts.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 01 - Core Concepts

Pods Containers

Welcome to the NZRT Wiki Podcast. Today we’re looking at Pods & Containers.

If you’re working with Kubernetes, the first concept you need to get comfortable with is the Pod. A Pod is the smallest deployable unit in Kubernetes. Think of it as a wrapper that holds one or more containers together, letting them share the same network and storage resources.

So what does that actually mean in practice? Every container inside a Pod shares the same IP address and port space. That means if you have two containers running side by side in the same Pod, they can talk to each other just by using localhost — no complicated networking required. It’s as if they’re running on the same machine.

One thing that’s really important to understand about Pods is that they are ephemeral. They’re not designed to be repaired when something goes wrong — they’re replaced. If a Pod fails, the system spins up a new one. Because of this, you never create Pods directly in a production environment. Instead, you manage them through something called a Deployment, which handles the lifecycle for you.

Now let’s talk about the Pod lifecycle. A Pod moves through a series of phases depending on what’s happening with its containers. There are five phases to know. First is Pending — this means the cluster has accepted the Pod, but the containers haven’t started yet. Next is Running, which means at least one container is currently active. Then there’s Succeeded, which means all containers finished their work and exited cleanly with a success code. After that is Failed — meaning at least one container exited with an error code, signalling something went wrong. And finally there’s Unknown, which means the system simply can’t determine the current state of the Pod. That one’s usually a sign of a communication issue between the node and the control plane.

Now let’s look at what a Pod definition actually looks like. If you were to write a basic Pod manifest — that’s the configuration file that tells Kubernetes what to create — it would specify a few key things. You’d declare the type of object you’re creating, in this case a Pod, and give it a name. In this example it’s called example-pod, placed in a namespace called nzrt-prod, with a label identifying it as part of an app. Then under the specification section, you’d define the container itself. Here the container is named app, it’s running an nginx web server at version 1.25, and it’s listening on port 80. You’d also set resource boundaries. In this example, the container requests a minimum of 100 millicores of CPU and 128 megabytes of memory, with a hard ceiling of 500 millicores of CPU and 256 megabytes of memory. Those limits make sure one misbehaving container can’t consume all the resources on your node.

Finally, let’s cover the commands you’ll actually use day to day when working with Pods. There are five core ones to know. The first lists all Pods running in your production namespace, giving you a quick overview of what’s up and what isn’t. The second lets you describe a specific Pod in detail — this is your go-to when something looks wrong, because it surfaces events, container statuses, and resource usage all in one place. Third is the logs command, which streams output from a named Pod so you can see exactly what the application inside is doing, or where it crashed. Fourth is the exec command, which drops you into an interactive shell inside a running Pod — think of it like connecting directly into the container to poke around. And fifth is the delete command, which removes a Pod by name. Just keep in mind that if that Pod is managed by a Deployment, a replacement will start up automatically right after you delete it.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 01 - Core Concepts

Namespaces

Welcome to the NZRT Wiki Podcast. Today we’re looking at Namespaces.

So, what is a namespace? Think of it as a virtual cluster living inside your actual Kubernetes cluster. Instead of one big shared space where all your workloads, configs, and resources sit together, namespaces let you carve things up into isolated sections. Each section can have its own access controls, its own resource limits, and its own set of rules. It’s a clean way to separate concerns — especially when you’re running multiple environments on the same cluster.

Let’s talk about how NZRT uses namespaces specifically. There are six you need to know about. First is nzrt-prod, which is your production environment. This one is locked down — only xc and dan have access, so treat it with care. Then there’s nzrt-staging, which mirrors the production config and is used for pre-production testing and UAT work. Below that sits nzrt-dev, a more relaxed space for development and experimentation — resource limits are looser here, so you’ve got room to move.

Beyond those three environment namespaces, there are a few infrastructure ones. The monitoring namespace is home to the shared observability stack — that’s Prometheus, Grafana, and Loki all running together. Then there’s ingress-nginx, which handles the Nginx ingress controller for the whole cluster. And finally, kube-system — this is where Kubernetes puts its own system components. Leave this one alone. Don’t modify it.

Now, Kubernetes also comes with a set of default namespaces out of the box. The one simply called default catches any resource that doesn’t have a namespace explicitly assigned to it. Kube-system, as mentioned, is for the core Kubernetes internals. Kube-public is readable by everyone in the cluster and holds general cluster information. And kube-node-lease is used internally for node heartbeat tracking — you won’t need to touch that one directly either.

Let’s walk through some of the common commands you’ll use day to day. If you want to see all the namespaces currently in your cluster, you run a get namespaces command with kubectl. To create a new namespace — say, nzrt-staging — you use kubectl create namespace followed by the name. If you want to see all pods running across every namespace at once, you pass the all-namespaces flag to a get pods command. To narrow it down to just one namespace, like nzrt-prod, you use the n flag followed by the namespace name. And if you want to set a default namespace for your current context so you don’t have to type it every time, there’s a config set-context command where you pass the current flag and then specify the namespace you want to default to.

Now let’s look at resource quotas, because this is where namespaces really earn their keep in a shared cluster. A resource quota is a Kubernetes object that caps how much compute a namespace can consume. In the example for nzrt-prod, the quota is named nzrt-prod-quota and it’s scoped to the nzrt-prod namespace. It sets a minimum CPU request of four cores, a minimum memory request of four gigabytes, a maximum CPU limit of eight cores, a maximum memory limit of eight gigabytes, and a hard cap of twenty pods running at any one time. This means no matter what gets deployed into production, it can never exceed those boundaries — which protects the rest of the cluster from being starved of resources by a runaway workload.

If you want to go deeper on access control within namespaces, check out the RBAC wiki page, which covers role-based access policies. And for the broader picture of how all of this fits together, the NZRT Kubernetes Architecture page is your next stop.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 04 - Networking

Networking Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Networking Overview.

Kubernetes networking is the system that connects everything inside your cluster — pods talking to other pods, services providing stable endpoints, and external traffic finding its way in. If you’ve ever wondered how all those moving pieces actually reach each other, this episode walks you through it.

Let’s start with the core networking model, because Kubernetes makes some strong guarantees here that are worth understanding upfront. First, every pod gets its own unique IP address that is valid across the entire cluster. Not just within a node — across the whole cluster. Second, any pod can communicate directly with any other pod without going through network address translation, which you might know as NAT. Third, nodes can also reach any pod directly without NAT. And finally, pod IP addresses are never masqueraded — meaning what you see is what you get, no hidden remapping happening underneath. These four rules together form what’s called the Kubernetes network model, and every networking plugin you install has to honour them.

Now, let’s talk about how traffic actually flows from the outside world into your application. Picture this as a journey with a few stops. It starts with an external client — that’s someone’s browser, a mobile app, whatever is making a request from outside the cluster. That request first hits what’s called an Ingress Controller. In this setup the Ingress Controller is running nginx. The Ingress Controller is the gatekeeper — it reads your routing rules and decides where to send the traffic next. From there, the request moves to a Service. Services in Kubernetes have what’s called a ClusterIP, which is a stable internal address that doesn’t change even if the pods behind it do. Think of the Service as a reliable middleman. Finally, the request lands at the actual Pod — the container running your application. So the journey is: external client, then Ingress Controller, then Service, then Pod. Four hops, clean and predictable.

Next up is the CNI, which stands for Container Network Interface. This is the plugin layer that actually implements the network model we just described. Kubernetes doesn’t hard-code a single networking solution — instead it lets you choose a CNI plugin that suits your needs. There are four common options you’ll encounter. The first is Flannel, which is simple to set up and uses an overlay network — a good starting point if you want something that just works without a lot of configuration. The second is Calico, which gives you support for network policies and BGP routing, making it a strong choice when you need fine-grained control over which pods can talk to which. The third is Cilium, which is eBPF-based and gives you advanced observability — if you need deep visibility into your network traffic and performance, Cilium is worth a look. The fourth is Weave Net, which is known for easy setup and comes with encrypted overlay networking out of the box, so your pod-to-pod traffic is protected without extra configuration.

Choosing the right CNI depends on your priorities — simplicity, policy control, observability, or security. Most production clusters end up on Calico or Cilium, but Flannel and Weave Net are perfectly valid for smaller or simpler environments.

Now let’s cover DNS, because once pods and services exist, you need a way to find them by name rather than memorising IP addresses. Kubernetes handles this through CoreDNS, which runs inside the cluster and answers name lookups automatically. Services get a DNS name that follows a pattern: the service name, then the namespace it lives in, then the suffix svc.cluster.local. So if you have a service called api in a namespace called production, you can reach it at api.production.svc.cluster.local. Pods also get DNS names, though they’re used less often. A pod’s DNS name uses its IP address with dashes instead of dots, followed by the namespace, then pod.cluster.local. The important thing to know is that CoreDNS makes service discovery automatic — your application code can use a stable DNS name rather than hardcoding IPs that might change.

To wrap up, here’s the big picture. Every pod has its own IP, pods talk to each other freely without NAT, and the network model is enforced by whichever CNI plugin you choose. External traffic enters through an Ingress Controller, passes through a Service, and reaches your Pod. CoreDNS keeps name resolution working so everything can find everything else by name. If you want to go deeper, the related topics to explore next are Services, Ingress, and Services and DNS — each of those builds directly on what we’ve covered here.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 06 - Security

Rbac

Welcome to the NZRT Wiki Podcast. Today we’re looking at RBAC.

RBAC stands for Role-Based Access Control, and at its core it answers three questions: who are you, what are you allowed to do, and where in the system can you do it? Whether you’re a human user, a service account, or an automated process, RBAC is what decides your level of access inside a Kubernetes cluster.

Let’s start with the four main building blocks of RBAC, because understanding these will make everything else click into place.

The first is a Role. A Role is scoped to a single namespace, and it defines what permissions are allowed within that one namespace. Think of it as a job description that only applies to one department.

The second is a ClusterRole. This works the same way as a Role, but instead of being limited to one namespace, it applies across the entire cluster. So if you need someone to have permissions everywhere, ClusterRole is your tool.

The third object is a RoleBinding. This is the glue between a Role and a user, group, or service account. On its own, a Role doesn’t do anything — you need a RoleBinding to actually hand those permissions to someone. And like a regular Role, a RoleBinding is scoped to one namespace.

The fourth is a ClusterRoleBinding. Same idea as a RoleBinding, but it applies a ClusterRole across the whole cluster, not just one namespace.

So the pattern is: Roles and RoleBindings live in a namespace. ClusterRoles and ClusterRoleBindings operate cluster-wide.

Now let’s look at a real example — a Role definition for NZRT developers. This configuration creates a Role called nzrt-developer, sitting inside the nzrt-dev namespace. It defines two sets of rules. The first set covers pods, deployments, services, and config maps — and it grants the ability to get information about them, list them, watch for changes, create new ones, update existing ones, and patch them. The second set is specifically for pod logs and interactive pod sessions — and for those, only get and create are permitted. So developers get solid read and write access to the main workload resources, but nothing beyond what they need.

Next up is the RoleBinding that actually activates those permissions. This binding is called dev-binding, also in the nzrt-dev namespace. It points to a specific user — the developer at nzrtnetwork dot com email address — and it references the nzrt-developer Role we just described. The moment this binding exists, that developer account gains all the permissions defined in the Role, but only inside the nzrt-dev namespace. Outside that namespace, those permissions simply don’t apply.

There’s also a third type of object worth knowing about: a ServiceAccount. This is used when an application or automated process — rather than a human — needs to interact with the cluster. The example here creates a ServiceAccount called nzrt-app-sa inside the nzrt-prod namespace. You’d then bind a Role or ClusterRole to this service account just like you would with a user, so the application running in your cluster has exactly the access it needs and nothing more.

Now let’s talk about how you actually work with these objects day to day. There are a handful of commands you’ll use regularly. You can list all Roles inside a specific namespace — say, nzrt-prod. You can do the same for RoleBindings in that namespace. You can pull a full list of all ClusterRoles across the entire cluster. And there’s a particularly useful command that lets you ask a direct question: can this specific user create deployments in this namespace? You pass in the namespace and impersonate the user in question, and Kubernetes will tell you yes or no. That last one is invaluable for debugging access issues without having to log in as the affected user yourself.

So to pull it all together: RBAC gives you fine-grained control over who can do what and where. You define permissions in a Role or ClusterRole, you attach those permissions to people or processes using bindings, and you use service accounts when the entity needing access is an application rather than a person. The scope — namespace versus cluster-wide — determines how broadly those permissions reach.

If you want to go deeper, check out the Security Overview and Namespaces articles in the wiki — both are closely related to everything we’ve covered here today.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S

Security Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Security Overview.

If you’ve spent any time working with Kubernetes, you’ll know that security isn’t just one thing you switch on. It’s a set of overlapping layers, and understanding how those layers fit together is what this episode is all about.

The core idea is this: Kubernetes security is built in depth. You’re protecting the cluster from multiple angles at the same time, covering who can access it, what traffic can flow through it, how workloads are isolated from each other, and how sensitive information is stored. Let’s walk through each of those layers.

The first layer is authentication. This is about proving who you are when you connect to the cluster. You might do that through a kubeconfig file on your local machine, through a ServiceAccount token assigned to an application running inside the cluster, or through an external identity provider using a protocol called OIDC, which stands for OpenID Connect. Think of authentication as the front door.

Once you’re through the front door, the next layer is authorisation. Kubernetes uses something called RBAC, which stands for Role-Based Access Control. This is how the cluster decides what you’re allowed to do once it knows who you are. You might be authenticated as a valid user, but RBAC is what determines whether you can actually create, read, modify, or delete specific resources.

The third layer is admission control. This is a set of gatekeepers that check requests before they’re written to the cluster. Tools here include LimitRange and ResourceQuota, which stop workloads from consuming too many resources, and a policy engine called OPA Gatekeeper, which lets you enforce custom rules across the cluster.

Layer four is the network. Kubernetes lets you define NetworkPolicies, which control exactly what traffic is allowed in and out of each pod. You can think of these as firewall rules at the pod level, giving you fine-grained control over ingress and egress traffic.

Fifth is pod isolation. This is handled through something called the SecurityContext, which is a set of settings you apply to a pod or container. At NZRT, the standard is to run containers as non-root users wherever possible and to use read-only filesystems. This limits the damage an attacker can do if they manage to get inside a running container.

The sixth and final layer is secrets management. Kubernetes Secrets store sensitive values like passwords, tokens, and API keys. The important thing here is that secrets should be encrypted at rest, meaning they’re not just sitting in plain text in the cluster’s data store. NZRT also uses external secrets managers for more advanced cases.

Now let’s talk about the NZRT-specific security baseline, because this is where those general principles get applied to our own environment.

The production namespace, which is called nzrt-prod, is restricted. Only the roles assigned to xc and dan have access to it. Secrets in that namespace are written using a format called stringData, and encryption at rest must be enabled on the cluster for this to be properly protected. All containers run as non-root where possible, as mentioned earlier. There’s also a ResourceQuota applied to nzrt-prod, which acts as a safety net to prevent any one workload from accidentally, or maliciously, consuming all available cluster resources. And ingress traffic into the cluster uses TLS, with certificates managed automatically by cert-manager using Let’s Encrypt.

Now let’s look at a few quick checks you can run to verify that security is configured correctly. These are command-line checks you’d run against the cluster.

The first check lets you ask the cluster a direct question: can a given user create pods in the nzrt-prod namespace? You’re essentially asking the cluster to tell you whether a particular action is permitted.

The second check is broader. You can ask the cluster to list all the things a user is allowed to do within the nzrt-prod namespace, giving you a full picture of their permissions.

The third check lets you inspect a specific pod and retrieve its security context settings, so you can confirm whether it’s running with the expected non-root and filesystem configurations.

And the fourth check simply lists all the NetworkPolicies that exist in nzrt-prod, so you can see at a glance whether traffic controls are in place.

None of these checks change anything in the cluster. They’re read-only inspection commands, which makes them safe to run at any time if you want to audit what’s in place.

To bring it all together, think of Kubernetes security as concentric rings. The outer ring is authentication and authorisation, controlling who gets in and what they can do. The middle rings are admission control and networking, shaping what workloads can run and how they communicate. The inner rings are pod isolation and secrets management, protecting the workload itself and the sensitive data it handles. NZRT’s baseline tries to get all of those rings working together, particularly in the production namespace where it matters most.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 04 - Networking

Services Dns

Welcome to the NZRT Wiki Podcast. Today we’re looking at Services & DNS.

Let’s start with the basics. Inside a Kubernetes cluster, CoreDNS is the engine that handles automatic DNS resolution. That means when a service or a pod needs to find another service or pod by name, CoreDNS is the one doing the lookup behind the scenes.

Now, DNS names in the cluster follow a predictable pattern depending on what you’re trying to reach and where you’re reaching from. There are four main formats to know. First, if you’re looking up a service from within the same namespace, you can just use the service name on its own — short and simple. Second, if you’re crossing into a different namespace, you add the namespace after the service name, separated by a dot. Third, if you want the full, unambiguous address — what’s called a Fully Qualified Domain Name — you build it out as the service name, then the namespace, then the suffix “svc.cluster.local”. And fourth, pods themselves get a DNS name too, built from the pod’s IP address with dashes instead of dots, followed by the namespace and then “pod.cluster.local”. A real example of a fully qualified service name would look like: wordpress-service dot nzrt-prod dot svc dot cluster dot local.

Next up, testing DNS resolution. The wiki shows two ways to do this. The first approach launches a temporary debug pod using a minimal image called busybox, runs an nslookup command against the wordpress-service inside the nzrt-prod namespace, and then automatically removes itself when done. The second approach skips creating a new pod altogether — instead, you jump directly into a pod that’s already running and fire the nslookup from inside it, this time using the cross-namespace format with the namespace included in the name.

Now let’s talk about headless services, because these work a little differently. A normal service gives you a single stable IP address that load-balances traffic across your pods. A headless service, by contrast, has no cluster IP at all — and that’s intentional. When you query a headless service, you get back the actual IP addresses of the individual pods behind it. This is particularly useful for StatefulSets, where each pod needs its own stable, predictable DNS name.

In the Nextcloud example from the wiki, you’d have two pods addressable individually. The first would be nextcloud-0 dot nextcloud dot nzrt-prod dot svc dot cluster dot local, and the second would be nextcloud-1 with the same suffix. Each pod is reachable directly by name, which matters a lot for stateful applications where pod identity is important.

The configuration for a headless service is straightforward. You define it as a standard Kubernetes service, give it the name “nextcloud” in the nzrt-prod namespace, set the cluster IP field explicitly to the word “None” — that’s what makes it headless — point the selector at pods with the app label of nextcloud, and expose port 80.

Finally, if you ever need to inspect or troubleshoot the CoreDNS configuration itself, the wiki shows a command that retrieves the CoreDNS config map from the kube-system namespace and prints it out in full. This is handy if you’re chasing a DNS issue at the cluster level and need to see exactly how CoreDNS is configured.

For more context, the related topics in the wiki are Networking Overview, Services, and StatefulSets — worth reading alongside this one if you want the full picture.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 01 - Core Concepts

Services

Welcome to the NZRT Wiki Podcast. Today we’re looking at Services.

So, what is a Service in Kubernetes? At its core, a Service gives you a stable network endpoint — think of it as a fixed address made up of a DNS name and an IP — for a group of pods. Here’s the thing about pods: they come and go. They spin up, they restart, they get replaced. But the Service address stays constant. That’s the whole point. You point your traffic at the Service, and Kubernetes takes care of routing it to whichever pods are actually running behind the scenes.

Now, not all Services work the same way. There are four types, and choosing the right one depends on what you’re trying to do.

The first type is called ClusterIP, and it’s the default. This one is for internal access only — pod-to-pod communication inside your cluster. Nothing outside the cluster can reach it directly.

The second type is NodePort. This exposes your service on each node’s IP address at a fixed port number. It’s a step up from ClusterIP in that you can reach it from outside the cluster, but it’s fairly manual and better suited for testing or simple setups.

Third is LoadBalancer. This one provisions an actual cloud load balancer — so if you’re running on a cloud provider, Kubernetes will automatically create a load balancer and wire it up to your service. This is your go-to for proper external access in production.

And the fourth type is ExternalName. Rather than routing to pods inside your cluster, this maps your service to an external DNS name. Useful when you need your cluster workloads to talk to something living entirely outside Kubernetes.

Let’s look at what a Service definition actually looks like. The example we have is a ClusterIP service for WordPress, sitting in the nzrt-prod namespace. The definition tells Kubernetes which pods to route traffic to — in this case, any pod labelled as a WordPress app. It then maps port 80 on the Service itself through to port 80 on those pods, using standard TCP. And since no type is explicitly set to something else, it defaults to ClusterIP, meaning internal only.

That’s a pretty typical pattern: you define which pods to target, you declare the port mapping, and you pick your type. Kubernetes handles the rest.

Next up, DNS. Once your Service is running, how do other things in the cluster actually talk to it? Kubernetes gives every service a DNS name that follows a predictable pattern. You take the service name, add the namespace, and then the suffix svc dot cluster dot local. So for the WordPress service in the nzrt-prod namespace, the full address would be wordpress-service dot nzrt-prod dot svc dot cluster dot local. That’s the fully qualified form.

If you’re working from within the same namespace, you can skip most of that and just use the service name on its own. Kubernetes DNS figures out the rest automatically.

Finally, let’s talk about the commands you’ll use most often when working with Services. There are three key ones. The first retrieves a list of all services running in the nzrt-prod namespace — handy for a quick overview of what’s live. The second gives you a detailed description of a specific service — in this case the WordPress service — including its selector, ports, and current state. And the third shows you the endpoints for that service, meaning the actual IP addresses and ports of the pods currently backing it. That last one is especially useful when you’re troubleshooting and you want to confirm that traffic actually has somewhere to go.

So to tie it all together: Services are how Kubernetes gives you a stable, reliable way to reach your pods no matter how many times they restart or get rescheduled. You pick the type that matches your use case — ClusterIP for internal, NodePort or LoadBalancer for external, ExternalName for mapping to outside systems. You write a short manifest, apply it, and from that point on your cluster has a consistent address to work with. DNS handles discovery automatically, and a handful of commands let you inspect and debug everything when you need to.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.

000K8S, 05 - Storage

Storage Overview

Welcome to the NZRT Wiki Podcast. Today we’re looking at Storage Overview.

If you’ve worked with Kubernetes before, you’ve probably noticed that it goes out of its way to separate your applications from the details of where their data actually lives. That’s the core idea behind Kubernetes storage — it abstracts the underlying storage system away from your workloads, so your pods don’t need to know whether their data is sitting on a local disk, a network drive, or a cloud volume. They just ask for storage, and Kubernetes handles the rest.

Let’s walk through how that actually works, because there’s a clear hierarchy to understand here. Think of it as a chain of four layers, each one sitting on top of the next.

At the top of that chain is something called a StorageClass. This is essentially a template or a definition that says “here’s the type of storage we can provision, and here’s how to provision it.” It points to a provisioner — the thing that actually knows how to create storage on your underlying infrastructure.

Beneath the StorageClass sits the PersistentVolume, or PV. This is the actual storage resource — the real chunk of space that exists in your cluster. It can be created manually by an administrator, or it can be created automatically and dynamically by Kubernetes using the StorageClass you just defined.

Below the PersistentVolume is the PersistentVolumeClaim, or PVC. This is how a pod makes a request for storage. Instead of pointing directly at a specific volume, a pod says “I need some storage with these characteristics” — and the PVC is that request. Kubernetes then matches the claim to an appropriate PersistentVolume.

And finally, at the bottom of the chain, you have the Pod itself. The pod mounts the PVC as a volume, and from that point on it just reads and writes files like normal. It never needs to know anything about the StorageClass or the underlying PersistentVolume that made it all possible.

So to put it simply — StorageClass defines how to make storage, PersistentVolumes are the actual storage, PersistentVolumeClaims are how pods ask for it, and Pods consume it.

Now let’s look at the five key storage objects you’ll encounter in Kubernetes. There are five of them, each with a distinct purpose.

First, the PersistentVolume — this is a cluster-level resource. It represents actual storage, and it can be provisioned manually by an admin or created dynamically when a claim comes in.

Second, the PersistentVolumeClaim — this operates at the namespace level. It’s the pod’s way of saying “I need storage,” and Kubernetes binds it to a matching PersistentVolume.

Third, the StorageClass — this is the provisioner definition. When dynamic provisioning is enabled, Kubernetes uses the StorageClass as a blueprint to create new PersistentVolumes on demand.

Fourth, the ConfigMap — this is how you inject non-sensitive configuration data into your pods. Think of it as key-value pairs that your application can read at runtime, without baking that configuration into your container image.

And fifth, the Secret — this is similar to a ConfigMap, but intended for sensitive data like passwords, tokens, and certificates. One important thing to know here: Secrets are base64-encoded by default, but they are not encrypted. Base64 is an encoding format, not a security measure — so if you need proper encryption at rest, that requires additional configuration in your cluster.

The last thing to understand is access modes — these define how a volume can be mounted across nodes in your cluster. There are three modes to know about.

ReadWriteOnce means only one node can mount that volume with read-write access at a time. This is the most common mode and suits most standard workloads.

ReadOnlyMany means the volume can be mounted as read-only across many nodes simultaneously. Useful when multiple pods need to read shared data but none of them need to write.

ReadWriteMany means many nodes can mount the volume with full read-write access at the same time. This requires storage backends that explicitly support it — not all do — so check your provider’s documentation before assuming this is available.

If you want to dive deeper, the related topics from this wiki page include Persistent Volumes, which goes into the full lifecycle of PV creation and binding, ConfigMaps and Secrets, which covers how to use those objects in practice, and StatefulSets, which is where storage gets especially interesting because StatefulSets are designed for workloads that need stable, persistent storage across restarts.

That’s it for this episode of the NZRT Wiki Podcast. Thanks for listening.