The Keeper of a Thousand Castles: Understanding Kubernetes

When the kingdom grows beyond a few castles, someone must decide where every banner flies.

A single container is easy to understand. You package an application, start it, expose a port, and watch it run. Add a database, cache, and supporting service, and Docker Compose can organize the small caravan well enough for development and modest deployments. The trouble begins when that caravan becomes a kingdom. Once dozens or hundreds of containers must remain available across multiple machines, the difficult question is no longer how to start them. The difficult question becomes who keeps track of everything after you stop watching.

Imagine a kingdom dotted with castles. Each needs guards, supplies, roads, and a banner identifying whom it serves. Some become crowded, others sit half empty, and occasionally an entire fortress disappears. A ruler trying to administer every castle personally would spend every waking hour deciding where soldiers belong and what should happen when something fails. The kingdom has outgrown manual administration even though no individual castle has become particularly complicated.

Kubernetes exists because containerized systems eventually create the same problem. It is a container orchestration platform that coordinates workloads across a collection of machines rather than requiring engineers to manage every container individually. You describe what the system should look like, and Kubernetes continually works to make the running environment resemble that description. Kubernetes is therefore not primarily a better way to start containers. It is a system for maintaining intent while the pieces carrying out that intent remain temporary.

The Kingdom Has Outgrown the Castle

In the previous stage of our journey, Docker Compose enabled multiple services to run together. We could describe an application, its database, supporting services, networking, and configuration in a single place instead of manually starting containers. Compose gives structure to a multi-container application, but it still assumes a relatively small world. We usually know which machine is running the application and can reason about individual containers when something goes wrong.

Production scale changes that relationship. Applications may require several copies of the same service across multiple machines to improve capacity and resilience. Machines fail, new releases replace old instances, and traffic sometimes demands more copies than it did yesterday. At that point, manually deciding where every container belongs becomes an operational burden. The problem has shifted from running containers to coordinating them.

Container orchestration addresses that shift. Kubernetes can schedule workloads, maintain the desired number of instances, replace failed workloads, connect services, and coordinate changes across the environment. The important lesson extends beyond Kubernetes itself. Once a system becomes large enough, the useful abstraction is no longer the individual machine or container. It becomes the application’s desired state.

Instead of saying that a particular container must run on a particular server, we can say that three application instances should exist and remain available. Kubernetes decides where to run them based on resources and constraints. Engineers describe the outcome while the platform handles routine placement and recovery. The kingdom stops relying on a single administrator to remember where every banner belongs.

The Keeper Watches the Desired State

Desired state is the idea that makes the rest of Kubernetes easier to understand. You describe how the system should look, and Kubernetes repeatedly compares that description with what currently exists. When actual state differs from desired state, controllers attempt to reconcile the difference. The platform is not merely executing instructions. It is continually working to preserve an intended condition.

Suppose an application requires three instances. A manual approach might start three containers and rely on monitoring and human intervention if one disappears. Kubernetes records the intention that three instances should exist. If a crash or machine failure leaves only two, the desired state remains three, so Kubernetes creates a replacement.

A simple Deployment expresses that intention:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: castle-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: castle-api
  template:
    metadata:
      labels:
        app: castle-api
    spec:
      containers:
        - name: api
          image: stacknscroll/castle-api:1.0
          ports:
            - containerPort: 8080

Notice what the definition does not contain. It does not identify three specific machines or provide a sequence for starting three containers. It declares that three replicas should exist. Kubernetes determines where they run and replaces them whenever reality drifts away from that declaration.

This difference between imperative action and declarative intent is one of the larger lessons hidden inside Kubernetes. An imperative system asks what action should happen next. A declarative system asks what should remain true. That makes recovery easier to automate, but it also places more responsibility on the declaration itself. When the desired state is incorrect, Kubernetes can preserve that state with remarkable consistency.

Human memory is an unreliable control plane.

Turning operational expectations into declarations makes them easier to review, version, reproduce, and enforce. That principle reaches far beyond Kubernetes. Reliable automation begins when important assumptions stop living only in runbooks, shell histories, and the memories of experienced engineers.

The Castle Within the Castle: Understanding Pods

Containers brought us into this series, but Kubernetes normally manages them through an abstraction called a Pod. A Pod is the smallest deployable unit Kubernetes schedules. It usually contains one primary application container, although tightly coupled containers can share a Pod when they genuinely need the same network and lifecycle.

The additional abstraction gives Kubernetes a unit around which it can organize execution, networking, storage, and lifecycle. Containers inside the same Pod share a network identity and can communicate through localhost. They may also share mounted storage. A Pod is therefore a small execution environment rather than simply another name for a container.

The most important characteristic of a Pod is that it should be treated as disposable. Kubernetes does not promise that one particular Pod will remain alive forever. If it fails, the platform may replace it with another Pod on another machine. Applications should not treat individual Pods like permanent castles whose addresses or local contents must survive.

That disposability is what gives orchestration much of its flexibility. Kubernetes can recover, move, scale, and update workloads because individual instances are replaceable. Systems become easier to operate when an application instance can disappear without taking irreplaceable knowledge with it.

Deployments Govern the Garrisons

Production applications are usually managed through higher-level resources such as Deployments rather than by creating individual Pods directly. A Deployment describes the workload Kubernetes should maintain, including how many replicas should exist. If a Pod disappears, the Deployment still requires the specified number of replicas. If the application image changes, Kubernetes can replace old Pods with new ones while preserving that intent.

The fantasy metaphor is useful here. A Pod is one garrison occupying one fortress. A Deployment is the royal decree stating that three garrisons must always defend the northern road. One fortress can fall without changing the decree. Kubernetes creates another because the system is organized around preserving the rule rather than preserving the individual instance.

This is why Kubernetes engineers learn to think less about individual containers and more about desired outcomes. The platform continually asks whether reality still matches the specification and acts when it does not. What looks like automatic healing is really repeated reconciliation. The physical machines still exist beneath the abstraction, but they no longer serve as the units around which the application is designed.

The castle remains real. It simply stops being the thing the kingdom depends on most.

The Roads Between the Castles: Services and Stable Identity

Pods are disposable, which creates an immediate networking problem. If Pods can disappear and return with different addresses, applications cannot safely depend on knowing where a particular Pod lives. The kingdom therefore needs something more stable than the individual fortress. It needs a permanent road sign.

In Kubernetes, that role is commonly filled by a Service. A Service provides a consistent network identity for a changing set of Pods. Clients connect to the Service rather than targeting specific Pods, and Kubernetes routes traffic to Pods that match the Service selector. The individual destinations may change, but the logical destination remains stable.

A simple Service might look like this:

apiVersion: v1
kind: Service
metadata:
  name: castle-api
spec:
  selector:
    app: castle-api
  ports:
    - port: 80
      targetPort: 8080

The relationship is created through labels. Our Deployment creates Pods labeled app: castle-api and the Service selects Pods carrying that label. It does not need to know which Pod names or IP addresses currently exist. Kubernetes keeps the connection between the stable Service and the changing Pods up to date.

This separation between identity and instance is one of the most useful architectural ideas Kubernetes teaches. The application known as castle-api is not one particular Pod. It is the capability represented by whichever healthy Pods currently fulfill that responsibility. Reliable distributed systems depend on this kind of stable abstraction because consumers should depend on the role being provided, not on the temporary worker currently providing it.

The Scheduler and the Empty Rooms

Once Kubernetes knows that several Pods should exist, it must decide where they run. A cluster contains worker machines, usually called nodes, with finite CPU, memory, and other resources. The scheduler evaluates Pods that need placement and selects suitable nodes for them. Engineers describe what a workload needs rather than manually assigning every instance to a machine.

Resource requests and limits help make those needs explicit:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Requests help Kubernetes decide where a workload can fit, while limits constrain how much of a resource a container may consume. The values still require judgment. Requests that are too low can encourage overcrowding, while values that are too high can strand usable capacity or prevent workloads from being scheduled.

The deeper lesson is that automation improves when assumptions become explicit. Kubernetes can replace tribal knowledge about which application belongs on which server with a repeatable placement process. The scheduler makes the placement decision, but engineers still provide the information that makes the decision meaningful.

When a Castle Falls

Failure is where orchestration begins to justify much of its complexity. If a node disappears, several Pods may vanish with it. In a manually managed environment, someone may need to determine what was lost, find spare capacity, and restart the affected workloads. Recovery depends on documentation, tooling, and the availability of the right engineer.

Kubernetes reacts differently because the desired state still exists. If a Deployment requires three replicas and only two remain, the platform knows reality no longer matches the declaration. It can create another Pod and schedule it on a healthy node without waiting for someone to manually reconstruct the lost workload.

That does not make Kubernetes a substitute for resilient application design. Replacement Pods can still depend on unavailable storage, overloaded databases, or failed services. The cluster may lack spare capacity, and an application may depend on state that vanished when the failed node was removed.

Kubernetes automates recovery from certain failures, but resilience still depends on the surrounding architecture. A fragile application deployed onto Kubernetes is still fragile. It may simply restart faster.

Health Is More Than Being Alive

A running process is not necessarily a useful process. It may be alive while unable to serve requests, complete initialization, or recover from an internal failure. Kubernetes therefore allows workloads to expose health signals that help the platform decide what to do next.

A liveness probe helps determine whether a container should continue running. A readiness probe answers a different question: whether that instance should currently receive traffic. An application can therefore be alive but temporarily unavailable while it loads configuration, warms a cache, or establishes dependencies.

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080

The mechanism is simple, but the engineering decision is not. A liveness check that fails during a brief outage of a remote dependency can cause Kubernetes to restart otherwise healthy instances. A readiness check that always reports success provides little protection.

Health checks are architecture expressed as operational signals.

The useful questions are therefore architectural ones. Is the process fundamentally broken? Can it safely receive traffic? Would restarting it improve anything? Kubernetes can automate the response, but engineers still have to define what the signals mean.

Changing the Garrison Without Closing the Gates

Orchestration must manage both change and failure. When a Deployment moves from one application image to another, Kubernetes can create new Pods while gradually retiring the old ones. Readiness checks help prevent new instances from receiving traffic too early, while the Service continues providing a stable destination throughout the transition.

This works because Pods are disposable. Kubernetes does not need to carefully transform an old instance into a new one. It can create a replacement representing the new desired state and remove the old instance when appropriate. The same idea appears in immutable infrastructure more broadly: replacement is often easier to reason about than repairing environments whose histories have become difficult to reconstruct.

The same desired-state model also supports scaling. Increasing a Deployment from three replicas to six causes Kubernetes to create additional Pods and find places for them to run. Autoscaling can adjust those counts based on metrics, but more application instances do not automatically increase the capacity of databases, queues, or downstream services.

Scaling one component can simply move the bottleneck.

Kubernetes makes horizontal scaling easier to operate. Architecture determines whether horizontal scaling actually helps.

The Control Plane Behind the Throne

All of this coordination requires a part of Kubernetes responsible for maintaining the cluster itself. The control plane stores cluster state and runs the components responsible for scheduling and reconciliation, while worker nodes provide the machines where application Pods run.

At a high level, the API server provides the interface through which cluster resources are managed. Cluster state is stored in etcd. Controllers work to reconcile actual and desired state, and the scheduler chooses locations for Pods that need nodes. Developers do not need to memorize every component before using Kubernetes, but they should understand that the orchestration platform is itself a distributed system.

That distinction matters because Kubernetes does not eliminate operational responsibility. Someone still has to secure, monitor, upgrade, and maintain the platform. Managed Kubernetes services can shift part of that work to a provider, raising an important architectural question: which responsibilities should the engineering team own and which should it delegate?

That question points directly toward the next layer of our kingdom. Before reaching the clouds, however, there is one final Kubernetes lesson to carry forward: automation does not replace engineering judgment. It makes the consequences of that judgment easier to repeat.

The Keeper Is Not the Kingdom

Kubernetes creates a useful layer between applications and the machines carrying them, which can make the infrastructure beneath the cluster feel almost invisible. Engineers apply manifests, Pods appear, Services route traffic, and failed workloads are replaced without anyone having to choose a particular server. Yet those machines still have finite CPU, memory, storage, bandwidth, security boundaries, geographic locations, and cost. Kubernetes governs infrastructure, but it does not eliminate its consequences. A cluster without enough capacity still cannot schedule new workloads, and a slow or unreliable network still affects everything built on top of it.

That distinction explains why Kubernetes is not automatically the right destination for every containerized application. A small system running comfortably on a few containers may gain little from adding a control plane, cluster networking, resource policies, rollout strategies, access controls, and another operational vocabulary. Kubernetes solves genuine coordination problems, but those problems should already exist before the solution is introduced into the architecture. Otherwise, a team can spend more effort operating the platform than the application ever required.

Complexity is still a cost, even when the complexity is well engineered.

The mature question is therefore not whether Kubernetes is powerful. It clearly is. The useful question is whether orchestration now removes more complexity than it introduces.

What the Keeper Cannot Decide

A sophisticated orchestrator can create the impression that architecture has been delegated to the platform. It has not. Kubernetes can keep three instances of a service running, but it cannot decide whether three instances are appropriate. It can expose a Service, but it cannot determine whether two components should communicate synchronously. It can restart a failing application indefinitely without recognizing that each restart repeats the same design problem.

State makes this limitation particularly clear. Stateful applications can run in Kubernetes, but databases and durable systems still require deliberate decisions about persistence, replication, consistency, backup, and recovery. Treating a database exactly like a stateless API simply because both happen to run inside containers confuses packaging with behavior. Kubernetes can manage where a workload runs, but it cannot redefine the workload’s fundamental properties.

Security and observability follow the same pattern. Kubernetes offers mechanisms for access control, secrets, network isolation, logging, metrics, and health reporting, but none of them configure themselves into a good architecture. A permissive cluster remains permissive, and a self-healing application that nobody can diagnose may simply hide its failures more efficiently. Automation can enforce good decisions consistently, but it can enforce bad decisions just as faithfully.

Kubernetes automates orchestration. Engineers still decide what deserves to be orchestrated and what rules should govern it.

A Small Kingdom Before a Great One

When engineers first learn Kubernetes, there is a natural temptation to build an empire immediately. Namespaces appear, ingress controllers arrive, autoscaling is enabled, policies multiply, and a simple application suddenly requires a map large enough for a royal cartographer. That exploration can be useful while learning the platform. It becomes dangerous when platform sophistication is mistaken for architectural maturity.

A better approach is to understand each Kubernetes abstraction by focusing on the problem it solves. Pods provide disposable execution units. Deployments maintain workloads and coordinate replacement. Services provide stable identity. Resource declarations help the scheduler make placement decisions. Probes provide the platform with information about whether an instance can safely continue running or accept traffic.

Once those relationships are understood, the YAML becomes less mysterious. Kubernetes resources are not arbitrary configuration fragments. They are declarations participating in a control system whose job is to keep reality aligned with intent. That perspective is more valuable than memorizing every field in every manifest.

It is also why copied configurations deserve suspicion. Replica counts, resource values, probe behavior, rollout strategies, storage choices, and networking rules all encode assumptions about the system. A configuration that works beautifully for one workload may be inappropriate for another.

Configuration is architecture with consequences.

The better learning strategy is to remember the questions behind the objects. What should remain true? Which components can be replaced? Which identities must remain stable? What happens when a node disappears? Where does state survive? How will the platform know whether an instance is safe to receive traffic?

Those questions will outlive Kubernetes.

The Price of Having a Keeper

The strongest argument for Kubernetes is also the strongest warning against adopting it casually. It can coordinate scheduling, recovery, networking, rollouts, scaling, and workload management through a consistent platform. For organizations operating many services across many machines, that consistency can eliminate enormous amounts of repetitive operational work. A shared platform can also give application teams a common way to deploy, inspect, and manage workloads.

But the abstraction itself must be understood. Engineers still need to know why Pods remain pending, why Services have no endpoints, why probes fail, why rollouts stall, and where resources are being consumed. Someone must think about upgrades, access control, networking, monitoring, policy, and capacity. Managed Kubernetes services can reduce that burden, but they do not eliminate the operating model.

For a large engineering organization, that trade may be excellent. A platform team can spread the cost of orchestration across many services and provide capabilities individual teams would otherwise have to build themselves. For a small application with modest operational needs, the same platform may consume engineering time that would be better spent elsewhere.

Good engineering is not measured by how sophisticated the infrastructure looks. It is measured by whether the chosen infrastructure addresses the system’s actual problems. Kubernetes should be adopted as an architecture because orchestration has become valuable enough to justify its cost, not because every respectable kingdom is expected to own one.

The Map Above the Kingdom

We began this chapter with containers that had multiplied beyond the point where individual management remained practical. Kubernetes answered by changing the level at which we operate. Instead of supervising individual containers, we describe workloads. Instead of preserving particular instances, we preserve desired state. Instead of depending on temporary addresses, we create stable service identities. Instead of rebuilding every failed workload manually, controllers reconcile the environment toward its declared design.

Pods, Deployments, Services, scheduling, probes, rolling updates, and scaling are important mechanisms, but they all support the same larger idea. Mature infrastructure becomes easier to operate when the system knows what should remain true and can respond when reality drifts away from that intention. Kubernetes turns container management into a continuing process of reconciliation rather than a collection of isolated operational commands.

The lesson worth carrying beyond Kubernetes is not a command, resource type, or YAML field. The larger a system becomes, the less practical it is to manage its individual pieces and the more important it becomes to manage the rules that govern those pieces. That is the architectural shift from running containers to orchestrating systems.

Our kingdom now has a keeper capable of placing workloads, replacing failed instances, maintaining stable identities, and enforcing a declared state across many machines. Yet we have carefully avoided one large question: where do all of those machines come from?

Kubernetes governs castles. Someone still owns the land beneath them.

Next week’s theme in The Kingdom in the Clouds is “Building Above the Clouds,” where the kingdom begins moving beyond infrastructure it directly owns. On Monday, we will climb into Renting the Wizard’s Tower: Understanding Cloud Infrastructure and examine what changes when compute, storage, platforms, and managed services belong to someone else.

The keeper has learned to govern a thousand castles, and now the kingdom must decide whether it needs to own the stones.