Simulating Kubernetes From Scratch: Inside Kubetopia's Cluster Engine

Kubetopia is a browser game where you play the on-call engineer for a cartoon 3D island town that happens to run on Kubernetes. Every building is a node, every glowing crate is a pod, and when a deployment starts crash-looping the townsfolk get visibly unhappy. You fix it by typing real kubectl commands into a console.

The obvious question is what sits behind that console. The answer is the part of the project I'm proudest of and the part nobody sees: a Kubernetes engine I wrote myself, in TypeScript, from scratch. No client-go, no API server, no container runtime, no real cluster hiding in a WebAssembly sandbox. Just a plain object holding the cluster state and a function that moves it forward one tick at a time.

This post is about how that engine is built — and why building it was the whole point.

Why simulate it instead of using the real thing?

Two reasons, and the second one matters more than the first.

Control. A real cluster is a wonderful thing and a terrible teacher for a game. It's slow to stand up, it costs money, it needs an account and a kubeadm incantation before the player learns anything, and you cannot reliably script "a node dies at 02:14, and forty seconds later marketing ships a bad image." A simulation gives me determinism, instant failures, and — most importantly — the ability to make a failure legible. When a pod in Kubetopia crash-loops, I know exactly why, so I can have the game's retired-SRE character nudge you toward the right command instead of leaving you to grep a real kubelet log.

There's a practical constraint underneath that too: the game runs entirely client-side, statically generated, with no backend. If the cluster were real, there'd be a server, a cluster per player, and a bill.

Learning. This is the real reason. You can read the reconciliation docs for a year and still hold a slightly wrong model of what a controller actually does. Writing one forces every hand-wave to become a line of code. What exactly happens to a pod when its node stops answering? How long before it goes away? Who deletes it — the deployment, the node, or something else? You can be vague about that while you're just using Kubernetes. You cannot be vague about it while you're implementing it, because the code will do something, and if that something is wrong, the game will visibly misbehave.

Implementing a system you want to understand is, in my experience, the highest-bandwidth way to learn it. Kubetopia is a game, but the engine is a study aid I happened to ship.

The one rule: no Kubernetes in the Kubernetes game

The engine had to be mine, top to bottom. I checked this claim against the code before writing it down, so here it is precisely: the simulation lives in four files under src/lib/k8s/, roughly 1,500 lines of TypeScript, and the only third-party import anywhere in that folder is the yaml package. Nothing else. No Kubernetes client, no API-server shim, no cluster-in-a-box.

The one library is deliberate. I wanted kubectl apply -f to parse manifests with real YAML semantics — anchors, multi-document files, the type coercion quirks, the lot — because "my toy parser accepted it but the real one wouldn't" is exactly the kind of lie a learning simulator must not tell. Everything the parser hands back, from validation to reconciliation, is code I wrote.

Note also what the engine does not import: React, Zustand, Three.js, Next.js, Firebase. It is framework-free TypeScript that would run just as happily in Node, a test harness, or a terminal.

Four layers, one direction

The project is deliberately split so that the interesting part stays independent of the pretty part. I wrote about this dependency rule at length in my architecture post, and Kubetopia is the same idea at a smaller scale:

src/lib/k8s/       ← the engine: pure, framework-free TypeScript
    types.ts         the shape of a cluster
    engine.ts        the control loop, queries, mutations
    kubectl.ts       the command interpreter
    manifest.ts      YAML parsing, validation, serialization

src/lib/levels/    ← content: missions as data (cluster builders, objectives, events)
src/store/         ← glue: one Zustand store owning a play session
src/components/    ← presentation: React UI + the Three.js town

dependency direction:  components → store → levels → engine
                       (the engine imports from none of them)

That direction is the whole design. The 3D town is a view of the cluster object and nothing more: it reads pods and nodes and draws crates and buildings. Swap Three.js for a plain HTML table and the simulation is unaffected. Delete the game entirely and the engine still runs — which is precisely what the test suite does.

State first: the cluster is one plain object

Before any behaviour, I had to decide what a cluster is. Real Kubernetes stores versioned objects in etcd behind an API server. I collapsed all of that into a single mutable document:

export interface Cluster {
  tick: number;
  nodes: K8sNode[];
  pods: K8sPod[];
  deployments: K8sDeployment[];
  services: K8sService[];
  configMaps: K8sConfigMap[];
  secrets: K8sSecret[];
  events: ClusterEvent[];
  /** Simulated container registry keyed by image ref. */
  registry: Record<string, ImageInfo>;
}

No namespaces, no resource versions, no watch streams. Just arrays. Every controller in the engine reads and writes this one object, which is a fair model of the real thing: in Kubernetes, controllers don't talk to each other either — they all talk to the shared state and react to what they find there.

The pod type is where the simulation's granularity actually lives:

export interface K8sPod {
  uid: string;
  name: string;
  owner?: string;          // the deployment that created it
  nodeName?: string;       // set by the scheduler
  phase: PodPhase;         // Pending | ContainerCreating | Running | ...
  ready: boolean;          // readiness is NOT the same as Running
  restarts: number;
  image: string;
  cpu: number;             // requested millicores
  memory: number;          // requested Mi
  labels: Record<string, string>;
  createdAtTick: number;
  phaseTicks: number;      // how long it has been in this phase
  backoff: number;         // remaining CrashLoopBackOff ticks
}

Two fields there carry most of the weight. phaseTicks makes the lifecycle a timed state machine rather than an instant transition — containers take time to create, probes take time to pass, terminating pods hang around. And ready being separate from phase encodes the single most under-appreciated fact in Kubernetes: Running and Ready are different things, and an entire class of production outages lives in the gap between them.

The control loop

The heart of the engine is one exported function. One tick is roughly one real second — the UI just calls it on an interval:

export function tick(cluster: Cluster) {
  cluster.tick += 1;

  /* 1 — Deployment reconciliation: desired vs actual, nothing else */
  for (const d of cluster.deployments) {
    const live = livePodsOf(cluster, d.name);
    if (live.length < d.replicas) {
      // create the missing pods
    } else if (live.length > d.replicas) {
      // terminate the newest extras
    }
  }

  /* 2 — Scheduler: bind Pending pods to a node that fits */
  /* 3 — Pod lifecycle: advance every pod's phase by one step */
  /* 4 — Reaper: remove pods that have finished terminating */
}

Writing step 1 is the moment Kubernetes clicked for me. The deployment controller has no memory and no plan. It does not know a pod died, or that you deleted one, or that a node caught fire. Every tick it asks one question — how many live pods do I have versus how many I want? — and takes one corrective step. That's level-triggered reconciliation, and it is why Kubernetes is robust in a way event-driven systems usually aren't: there is no event to miss, because nothing is ever "handled". The loop just keeps closing the gap.

It's also why the answer to "how do I fix this pod?" in Kubernetes is so often "delete it." The controller doesn't repair pods. It counts them.

The scheduler: filter, then score

Real scheduling is a plugin framework with dozens of predicates. Mine is the same two-phase shape, shrunk to the parts a player can reason about — filter the nodes that can take the pod, then rank the survivors:

function schedulable(cluster: Cluster, pod: K8sPod, node: K8sNode): boolean {
  if (node.status !== "Ready" || node.cordoned) return false;
  if (node.roles === "control-plane") return false;
  const use = nodeUsage(cluster, node.name);
  return use.cpu + pod.cpu <= node.cpuCapacity
      && use.mem + pod.memory <= node.memCapacity;
}

Then candidates are sorted least-allocated-first, and the winner gets bound. Implementing this taught me something I had genuinely misunderstood: the scheduler does its arithmetic on requests, not on actual usage. A pod that requests 2 CPUs and uses none still occupies 2 CPUs of schedulable capacity. Once your engine does that math out loud, "why is my pod Pending when the nodes look idle?" stops being a mystery forever.

There's one deviation from the real thing that the game forced on me. Pods are sorted so that workloads marked critical claim free capacity first — a stand-in for pod priority. Without it, one mission became unwinnable: an analytics deployment that had scaled up earlier would grab every scrap of capacity freed by the payments app's own terminating pods, and a correct rollback would never recover. The player did everything right and the town still died. That bug taught me more about why priority and preemption exist than any doc page had.

The pod lifecycle

Step 3 is a switch over the pod's phase, and it's where all the interesting failure modes live:

Pending ──scheduled──► ContainerCreating ──image ok?──► Running ──probe──► Ready
                              │ no                          │
                              ▼                             │ crashes, or a
                       ImagePullBackOff                     │ missing config key
                              │ retry every 6 ticks         ▼
                              └──────────────────────► CrashLoopBackOff
                                                            │ backoff expires
                                                            └──► ContainerCreating

node goes NotReady ──4 ticks──► Unknown ──45 ticks──► evicted

The timings are tuned for a game (a container creates in 2 ticks; backoff is 4 + 2 × restarts capped at 20 ticks rather than true exponential growth, because a player staring at a five-minute backoff is not learning anything). But the shape is faithful, and the last line is the one I'd never internalised properly. When a node stops reporting, its pods are not deleted. They freeze, they go Unknown, and only after a long grace period are they evicted and rescheduled. Forty-five ticks of dead service. That gap is the entire reason kubectl drain exists, and I only truly felt it once I'd written the timer myself and watched a mission's happiness meter bleed out while waiting.

Failure modes are data, not code

An early temptation was to hard-code incidents: a function that "makes the bakery crash." I'm glad I didn't. Instead, every failure is a property of the cluster's data, and the same generic loop produces it. The simulated registry is just a map:

c.registry = {
  "ticket-shop:3.2": { exists: true, logs: ["listening on :8080"] },
  "poster-maker:2.0": { exists: true },
  // poster-maker:2.1 is deliberately absent — the intern fat-fingered the tag
  "band-website:1.0": { exists: true, crashes: true },
};

c.services.push(
  // the typo that killed ticket sales: this selector matches no pod
  { name: "ticket-svc", selectorApp: "tikcet-shop", port: 80 },
);

An image that isn't in the registry produces ImagePullBackOff. An image flagged crashes produces CrashLoopBackOff. A deployment referencing a ConfigMap or Secret key that doesn't exist crashes at startup. A readiness probe pointed at a port the container doesn't listen on produces the cruellest bug of all — a pod that runs forever and never becomes Ready. All of those funnel through one diagnostic function:

export function podStartupIssue(cluster: Cluster, deploymentName?: string): StartupIssue {
  const d = cluster.deployments.find((x) => x.name === deploymentName);
  if (!d) return {};
  const issue: StartupIssue = {};
  // missing ConfigMap key? missing Secret key? probe port ≠ container port?
  return issue;
}

One function, two consumers. The control loop calls it to decide whether a pod should crash or stay un-Ready; kubectl describe calls it to explain why. That's what keeps the game honest: the diagnosis the player reads is computed from the same source as the misbehaviour they're watching, so the console can never tell them something the simulation doesn't actually believe.

kubectl as an interpreter

The console is a single function with a deliberately boring signature:

runKubectl(cluster: Cluster, raw: string, files: Record<string, string>): CmdResult

export interface CmdResult {
  output: string;
  ok: boolean;
  /** the command wants the UI to open the YAML editor */
  editor?: EditorRequest;
}

Text in, text out, cluster mutated in place. There's no command registry or plugin system — real kubectl's grammar is irregular enough per verb (scale takes --replicas=, set image takes an assignment, patch takes JSON) that a switch statement over verbs is honestly the clearest thing. Around it sit the small details that make it feel real: alias tables so po, deploy and svc work, a kindAndName helper that accepts both deployment/web and deployment web, -o wide, a column formatter that pads to the widest cell, and -n default silently tolerated so the muscle memory of real users doesn't get punished.

The error strings are copied from the real thing on purpose — Error from server (NotFound): pods "web-abc12" not found — because half of learning kubectl is learning to read its errors. And describe secret prints key names and byte counts but never values, exactly like the original, because teaching a bad habit in a teaching tool would be unforgivable.

My favourite detail in this file is what happens with kubectl edit deployment/web. The interpreter does not open a modal — it has never heard of React. It serializes the live resource to YAML and returns an EditorRequest describing the UI it would like. The store decides what to do with that. The engine stays a pure data machine, and everything visual stays on the other side of the boundary.

YAML that fights back

manifest.ts is where the simulator earns its keep as a study tool, because writing YAML that Kubernetes rejects is the most universal beginner experience there is. Parsing is real; validation mirrors the API server's actual invariants. Non-integer replicas, RFC 1123 name rules, quantity formats like 250m and 256Mi, required fields — and the classic:

$ kubectl apply -f admissions.yaml
The Deployment "admissions-web" is invalid: spec.selector
"matchLabels.app=admissions-desk" does not match template labels
"app=admissions" — the selector must match the pod template's labels

The manifest layer works in both directions. It parses YAML into cluster resources for apply, and serializes resources back into YAML for edit — with a mustMatch guard so an edit session can't quietly change the kind or name of the thing it opened, which is what real kubectl does too. Applying a manifest whose pod spec changed triggers a rollout, so editing an image in the YAML behaves the same as kubectl set image. Same state, two roads to it.

The API the engine exposes

Because the rest of the app can only reach the cluster through this surface, keeping it small and boring was a design goal. In full:

// construction — used by mission definitions to build a starting world
emptyCluster(): Cluster
makeNode(name, opts?): K8sNode
makeDeployment(name, opts?): K8sDeployment

// time — the only way the world moves
tick(cluster): void

// queries — read-only; used by objectives, scoring and the 3D scene
podsOf / livePodsOf / readyPodsOf (cluster, deployment): K8sPod[]
nodeUsage(cluster, nodeName): { cpu, mem }
serviceHealth(cluster, serviceName): number      // 0..1
podStartupIssue(cluster, deployment): StartupIssue

// mutations — used by kubectl and by scripted mission events
createPodFor / terminatePod / rolloutDeployment
failNode / recoverNode / addEvent

// commands and manifests
runKubectl(cluster, raw, files): CmdResult
applyYaml(cluster, text, mustMatch?): string     // throws ManifestError
deploymentToYaml / serviceToYaml / configMapToYaml / secretToYaml

The entire game consumes that through four import lines in one file — the Zustand store — plus a single type-only import in the 3D layer so a building knows what a pod looks like. Everything else in the application talks to the store, never to the engine.

Two of those queries do more work than their size suggests. serviceHealth returns the fraction of a service's desired replicas that are actually Ready behind a selector that actually matches — which means a mistyped selector reads as zero health even though every pod is perfectly fine. The store averages that across services, weights critical workloads double, and eases the town's happiness meter toward it. The cartoon townsfolk are, quite literally, a rendering of endpoint availability.

And because the engine exposes queries rather than events, mission objectives can be written as predicates over cluster state instead of scripts:

{
  id: "fix-selector",
  title: "Reconnect the ticket booth",
  points: 250,
  check: (ctx) => {
    const svc = ctx.cluster.services.find((s) => s.name === "ticket-svc");
    return svc?.selectorApp === "ticket-shop"
        && deploymentHealthy(ctx.cluster, "ticket-shop", 2);
  },
}

An objective completes when the cluster genuinely reaches a healthy state, by any route the player invents. You cannot pass a mission by typing the magic command; you pass it by fixing the cluster. That property falls straight out of the layering — the checker and the simulation share one source of truth.

The Linux commands, and where the boundary sits

A console that only spoke kubectl would feel wrong. Real cluster work is interleaved with shell work: you ls the manifests, you cat one to see what you're about to apply, you clear the screen. So a handful of Unix commands are simulated too — and where they live is itself an architectural decision.

They are not in the engine. The engine owns the cluster; it has no business owning a filesystem. Instead the store intercepts them before the input ever reaches runKubectl:

if (trimmed === "clear")      { /* wipe the terminal buffer */ }
if (trimmed === "ls")         { /* list the mission's blueprint files */ }
if (trimmed.startsWith("cat ")) { /* print one, or: cat: x: No such file */ }
if (trimmed === "help")       { /* the command reference */ }
if (trimmed === "hint")       { /* nudge toward the next objective */ }
// otherwise → runKubectl(cluster, trimmed, files)

The "filesystem" is a flat map of filename to YAML text, handed to each mission as data:

files: {
  "admissions.yaml": ADMISSIONS_YAML,
}

That one map serves three commands. ls lists its keys, cat prints a value, and kubectl apply -f looks up a filename in it — reporting error: the path "x.yaml" does not exist. Files available: ... when you typo, which is a far kinder failure than the real one. When you fix a blueprint in the editor and apply it, the corrected text is written back into the map, so the file stays fixed for the rest of the mission, exactly like saving to disk. The terminal component adds the shell ergonomics on top: a $ prompt, arrow-key history, and focus handling so you can keep typing after a story pop-up.

I'll be honest about the scope, because a simulator that oversells itself is worse than useless: this is a facade, not a shell. There are no pipes, no redirection, no globbing, no working directory, no processes. ls takes no arguments and cat takes exactly one filename. The engine's kubectl layer is a genuine attempt at fidelity; the Linux layer is a set of props sized precisely to the lesson. Adding grep and output piping is the obvious next step, and it would mean building an actual token pipeline rather than a chain of string checks — a fair amount of work for a modest gain, so it waits.

There's a small nicety in the same spirit: the console accepts k as a shortcut for kubectl, so the objective checkers normalise a leading k back to kubectl before matching. Small thing. It's the kind of detail that decides whether a simulation feels like the real tool or like homework.

How do you test a simulation?

This turned out to be the most valuable thing I built after the loop itself. npm test plays every mission headlessly — no browser, no React, no WebGL — driving the real engine, the real kubectl interpreter and the real objective checks through the same sequence the game uses: engine tick, then scripted events, then objective evaluation.

Each mission is played twice, under two timing profiles:

• HAPPY — the player reacts the moment each incident lands.
• SLOW — every scripted event fires before the player acts at all.

The slow profile exists because it has caught genuinely unwinnable states twice. In one mission a node auto-recovered before a dawdling player could observe the failure, permanently invalidating a "detect the outage" objective. In another, the capacity land-grab I mentioned earlier starved a late rollback forever. Both were invisible in normal play and both were fatal if you hit them.

None of that is testable without the layering. Because the engine imports no framework, a mission is a pure function of its starting cluster and a list of commands — so a headless run is the game, not an approximation of it.

What implementing it taught me that using it never did

• Reconciliation is level-triggered. Controllers compare desired to actual and take one step. They don't handle events, so there's no event to lose.
• Pods are cattle, structurally. The deployment controller has no repair logic at all. It creates and it deletes. That's the whole reason "just delete the pod" works.
• Running ≠ Ready. Two independent axes. Services route to Ready pods, so a perfectly healthy-looking Running pod can be serving nobody.
• Scheduling is arithmetic on requests. Not on usage. Unschedulable pods are usually a spreadsheet problem, not a mystery.
• A dead node doesn't delete anything quickly. Pods freeze, go Unknown, and are evicted only after a long grace period — which is exactly why you drain a node before you take it down, rather than after.
• Nothing is atomic. Every operation is convergence over time. Writing the timers made that visceral in a way reading about it never did.

What I deliberately left out

There are no namespaces, no RBAC, no StatefulSets or DaemonSets, no Ingress, no real networking, no split between kubelet and control plane, and no true rolling update — a rollout terminates the old pods and lets reconciliation bring up new ones, rather than honouring surge and unavailability budgets.

Every one of those is a deliberate omission, not a to-do. A simulator's job is fidelity where the lesson is, and each extra concept is a tax on every mission that doesn't need it. When a mission comes along that genuinely teaches rolling-update strategy, that's when the loop grows a rollout controller — and not one tick earlier.

Was it worth it?

Reimplementing a system to learn it sounds indulgent, and for a while I wondered whether I was building a game or an elaborate way to procrastinate on studying. It was the second one, and it worked far better than the first approach would have. Every simplification I made was a decision I had to justify, and justifying a simplification requires understanding the thing you're simplifying. The bugs were the best part: an unwinnable level taught me pod priority, and a mission that hung on a dead node taught me eviction timeouts, and neither lesson has faded the way reading would have.

So the engine ended up being both. It's the machinery that makes a small 3D town break in interesting ways, and it's the most effective set of Kubernetes notes I've ever written — except they run.

The fastest way I know to understand a system is to build a small, wrong version of it, then keep finding out exactly how it's wrong. 🏙️