Kubernetes

Part I: The mental model

kubectl apply (desired state, YAML)
        │
        ▼
API server ── authn → authz → admission → validation
        │
        ▼
etcd (the only state that exists)
        │  watch streams fan out to everyone
        ├──────────────┬────────────────┬───────────────┐
        ▼              ▼                ▼               ▼
 deployment ctrl   replicaset ctrl   scheduler       kubelet (per node)
 Deployment→RS     RS→Pods           Pod→Node        Pod→containers
        │              │                │               │
        └──── every arrow is a write BACK through ──────┘
              the API server, never a direct call
                                                        │
                                                        ▼
                                          CRI → containerd → Linux

Kubernetes is a database with opinions. Every object you have ever typed into YAML, pods, deployments, nodes, secrets, services, is a record persisted in etcd, a Raft-based consistent key-value store, and the API server is the sole process allowed to touch it. Everything else in the system, the controllers, the scheduler, the kubelet on every node, even kubectl, is a client of that one REST API. No Kubernetes component ever talks to another component directly; all coordination happens by reading and writing shared state through the API server. That single sentence is the architecture.

The second half of the identity is what the clients do with that state. Each controller runs a reconcile loop: watch the desired state written in objects' spec fields, observe the actual state of the world, and act to converge one toward the other, recording observations in status. The deployment controller reconciles Deployments into ReplicaSets, the replicaset controller reconciles ReplicaSets into Pods, the scheduler reconciles "pods with no node" into "pods bound to a node," and the kubelet reconciles "pods bound to my node" into running containers. Four different programs, one identical shape.

So the one-sentence identity: Kubernetes is a strongly consistent database of desired state plus a swarm of level-triggered reconcile loops that independently push reality toward it. The codebase is enormous, but it is enormous the way a crystal is large: the same structure repeated. Learn the write path once and the loop shape once, and you have learned the whole design. That is exactly what Parts IV and V do.

Part II: Using it

You do not install the kubernetes/kubernetes repository; you run a distribution of it. For a laptop, the standard choice is kind, which boots a fully conformant cluster inside Docker containers (minikube is the equally standard alternative). You also need kubectl, the CLI client. On Linux:

# kind (check kind.sigs.k8s.io for the current release; v0.32.0 as I write this)
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind

# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/kubectl

On macOS both come from Homebrew:

brew install kind kubectl

Creating a cluster takes about a minute. The default cluster is a single Docker container named kind-control-plane running the API server, etcd, the controller manager, the scheduler, and a kubelet all at once:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.xx.y) ...
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ...
$ kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   40s   v1.xx.y

(Versions in output vary with your kind release; everything in this chapter is version-stable behavior.) Now the first real workload. Write this to web.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
        ports:
        - containerPort: 80

$ kubectl apply -f web.yaml
deployment.apps/web created
$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-5b7f9c86d4-8kx2p   1/1     Running   0          15s
web-5b7f9c86d4-m4tqz   1/1     Running   0          15s
web-5b7f9c86d4-vw6r8   1/1     Running   0          15s

The random-looking middle segment (5b7f9c86d4, yours will differ) is the pod-template hash: the Deployment created a ReplicaSet named web-<hash>, and the ReplicaSet created pods named after itself. That naming chain is the object ownership chain, and Part IV follows it in the source.

From code, the natural language is Go, because the official client library, client-go, is the same one every controller in the ecosystem is built on. Listing pods takes ten lines:

config, err := clientcmd.BuildConfigFromFlags("",
    filepath.Join(homedir.HomeDir(), ".kube", "config"))
clientset, err := kubernetes.NewForConfig(config)

pods, err := clientset.CoreV1().Pods("default").
    List(context.Background(), metav1.ListOptions{})
for _, p := range pods.Items {
    fmt.Println(p.Name, p.Status.Phase)
}

Now the mistakes everyone makes at the start, because each one teaches a piece of the model. Mistake one: creating bare pods. kubectl run mypod --image=nginx creates a Pod with no controller above it. Delete it and it is gone forever; the self-healing you came for lives in the controller, not the pod:

# wrong: nothing will ever recreate this
kubectl run mypod --image=nginx

# right: a Deployment owns and replaces its pods
kubectl create deployment mypod --image=nginx

Mistake two: fighting the controller by editing its children. If you edit or delete a pod that a ReplicaSet owns, the ReplicaSet puts it back the way the spec says. Beginners read this as the cluster being haunted. It is the reconcile loop doing its one job, and the fix is always to edit the object at the top of the ownership chain, the Deployment.

Mistake three: the selector and the labels must match. In hand-written YAML, spec.selector is how the Deployment finds its pods and spec.template.metadata.labels is what the pods are stamped with. If they disagree, the API server rejects the object at validation time with selector does not match template labels, which is your first sighting of the validation stage from Part IV.

Mistake four: expecting latest to roll out. A Deployment only acts when its pod template changes. If your image tag is latest and you push a new image, the template text is identical, so nothing happens. Use immutable tags and change them explicitly:

# wrong: template unchanged, no rollout, and running pods
# silently diverge in version from future pods
image: myapp:latest

# right: every release is a visible spec change
kubectl set image deployment/web nginx=nginx:1.28

Mistake five: omitting resource requests. The scheduler places pods by comparing requests against node capacity. A pod with no requests fits anywhere on paper, so nodes overcommit until real memory runs out and the kernel OOM-kills whatever is largest. Requests are not optional metadata; they are the scheduler's input data.

Part III: When it is the right tool

Kubernetes earns its complexity when you have many services, more than one team, more than a handful of machines, and a need for the things it makes uniform: declarative deploys and rollbacks, self-healing, horizontal scaling, service discovery, and one API that every tool in the ecosystem (Helm, Argo, Prometheus, operators) can build on. It is also the right substrate when you want extensibility: CRDs and controllers let you make the platform manage databases, ML training jobs, or certificates with the same machinery that manages pods. If you serve a FastAPI application behind an nginx ingress, Kubernetes is the layer that decides where those processes run and keeps them running.

The honest alternatives. For a single small app, a VM with Docker Compose and a reverse proxy is simpler and fine. For "just run my container, bill me per request," serverless platforms like Cloud Run or AWS Fargate remove the cluster entirely and are the better default for small teams. HashiCorp Nomad is the credible orchestrator for shops that want scheduling without the Kubernetes API surface, and Amazon ECS is the pragmatic choice if you are all-in on AWS and want less to operate. The pattern in all four: they trade away the extensible API machinery, which is precisely the thing you should not pay for until you need it.

The architecture-shaped warning, this system's "writable SQLite on NFS": never build a second source of truth beside etcd, and never mutate cluster state out of band. The reconcile loops assume the API server's view is the truth. The moment a deploy script SSHes into nodes to restart containers, or an external system holds "the real list of what should be running" and pushes imperative commands, you have two authorities, and the controllers will fight the other one forever, reverting its changes on every sync. The correct integration direction is always to write desired state into the API (a GitOps tool like Argo CD does exactly this) and let the loops do the acting.

SAFE                                DANGEROUS
git repo ──► Argo CD ──► API server   deploy script ──ssh──► node
                          │  ▲                                 │
                    controllers        controllers see drift ◄─┘
                     converge          and revert it, forever

Part IV: The full life of one request

The canonical operation is the one you already ran: kubectl apply -f web.yaml creating a Deployment. It is the best single thread through the codebase because it visits every component. All file paths below are real paths in kubernetes/kubernetes, verified against the master branch as of mid-2026; the architecture they implement has been stable for a decade.

Stage 1: kubectl builds an HTTP request

kubectl is a thin client; its code lives in the main repo under staging/src/k8s.io/kubectl/ (published separately as kubernetes/kubectl). Apply parses your YAML, asks the API server's discovery endpoints what "apps/v1, Deployment" maps to as a REST path, and sends the object to /apis/apps/v1/namespaces/default/deployments. Modern apply is server-side apply: the server, not the client, merges your fields with the live object and records which manager owns which fields (managedFields), so two tools can co-own one object without clobbering each other. The key point: after this stage, kubectl's job is done. Nothing downstream knows or cares that a human was involved.

Stage 2: the API server gatekeeps: authn, authz, admission

The generic request machinery lives in staging/src/k8s.io/apiserver/. Every request passes a fixed gauntlet. Authentication establishes who you are (client certificates, bearer tokens, OIDC). Authorization asks whether that identity may perform this verb on this resource, almost always answered by RBAC rules, which are themselves just objects in etcd. Then admission: a chain of plugins that may mutate the object (fill defaults, inject sidecars) and then validate it against policy (quotas, security restrictions). The built-in plugins are readable under plugin/pkg/admission/, with resourcequota and limitranger as good first reads, and the same interface is exposed to you as mutating and validating webhooks, which is how tools like Istio and cert-manager hook the write path without forking Kubernetes.

Stage 3: validation and the registry

Each resource type has a strategy object defining its lifecycle rules: schema validation, field defaulting, what may change on update. For Deployments this is pkg/registry/apps/deployment/strategy.go, and the selector-mismatch error from Part II is thrown here. The registry layer (pkg/registry/) is the bridge between the generic REST handlers in staging/src/k8s.io/apiserver/pkg/endpoints/handlers/ (create.go, update.go, patch.go, watch.go) and typed storage: the handlers are the same for every resource, and the strategy is the per-type personality.

Stage 4: the etcd write

The storage backend is staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go. The Deployment is serialized (as protobuf, not JSON) and written to a key like /registry/deployments/default/web inside etcd, guarded by a transaction so concurrent writers cannot silently overwrite each other. etcd's revision counter for the key becomes the object's resourceVersion, which is Kubernetes' entire concurrency story: every update must present the version it read, and a stale version gets a 409 Conflict instead of a lock. Once etcd's Raft quorum commits the write, the request is durable and the API server returns 201. At this instant, zero containers exist. "Created" in Kubernetes means "recorded," and everything after this stage is asynchronous convergence.

Stage 5: the watch fan-out

Everything downstream is driven by watches. etcd streams key changes to the API server (staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go), and the API server's watch cache (staging/src/k8s.io/apiserver/pkg/storage/cacher/) maintains one in-memory copy per resource type and fans events out to every client watch, so a thousand watchers cost etcd one stream. A watch is just a long-lived HTTP request whose response body is a stream of {type: ADDED|MODIFIED|DELETED, object} events; you can see it raw with kubectl get deployments --watch -o yaml. The deployment controller has such a watch open, and our new object arrives on it within milliseconds.

Stage 6: the deployment controller creates a ReplicaSet

All the workload controllers run inside one binary, the kube-controller-manager. The deployment controller lives in pkg/controller/deployment/: deployment_controller.go wires informer events into a workqueue, sync.go holds the reconcile logic, and rolling.go and rollback.go implement rolling updates and rollbacks. Its reconcile compares the Deployment's pod template against existing ReplicaSets, finds none matching, and creates one named web-<pod-template-hash> with an ownerReference pointing back at the Deployment. On a later image change, the same code creates a second ReplicaSet and walks replicas between old and new under the maxSurge/maxUnavailable budget; a rollback is just re-inflating an old ReplicaSet, which is why revision history is stored as scaled-to-zero ReplicaSets. Note what "creates" means: another POST through the full gauntlet of stages 2 to 4. Controllers get no back door.

Stage 7: the replicaset controller creates Pods

The new ReplicaSet arrives on the replicaset controller's watch. pkg/controller/replicaset/replica_set.go is the single best controller to read in the whole repository: its job statement is one sentence (make the number of pods matching the selector equal spec.replicas), and its reconcile is literally a count, a subtraction, and a batch of creates or deletes. Two production details are worth noticing in the code: it creates pods in growing batches (1, then 2, then 4...) so a typo asking for ten thousand pods fails fast, and it tracks in-flight creations in an "expectations" cache so it does not double-create while its own watch events are still catching up to what it just did. Our three pods are now objects in etcd with spec.nodeName empty.

Stage 8: the scheduler binds each Pod to a node

The scheduler watches for exactly that shape: pods with no node. Its loop lives in pkg/scheduler/schedule_one.go, and each pod goes through the scheduling framework, a pipeline of plugin extension points: PreFilter and Filter plugins eliminate nodes that cannot host the pod (insufficient resources, taints, affinity violations), Score plugins rank the survivors (spread across nodes, prefer nodes that already have the image), and the highest scorer wins. The plugins are self-contained and readable under pkg/scheduler/framework/plugins/: noderesources, nodeaffinity, tainttoleration, interpodaffinity, podtopologyspread. Binding is, once again, just an API write: the scheduler POSTs a Binding to the pod's binding subresource, which sets spec.nodeName. The scheduler never contacts a node in its life. Part V returns to the framework in detail.

Stage 9: the kubelet makes it real

The kubelet on the chosen node watches the API server for pods with spec.nodeName equal to its own name, the only filtered watch in the story. pkg/kubelet/kubelet.go contains syncLoop, the node-level reconcile loop, which dispatches each pod to a per-pod worker (pod_workers.go). The worker drives the Container Runtime Interface, a gRPC API defined in staging/src/k8s.io/cri-api/ and implemented by containerd or CRI-O: create the pod sandbox (network namespace, IP address via CNI), pull nginx:1.27 through the ImageService if not present, then CreateContainer and StartContainer through the RuntimeService, with the translation logic in pkg/kubelet/kuberuntime/. The kubelet is the first and only component in this entire story that touches a container runtime.

Stage 10: status flows back up the same channels

Convergence is only half the loop; observation is the other half. The kubelet's PLEG (pod lifecycle event generator, pkg/kubelet/pleg/) notices container state changes and the status manager (pkg/kubelet/status/) writes status.phase: Running and the pod's IP back to the API server. That write lands in etcd, fans out on watches, and climbs the ownership chain in reverse: the replicaset controller sees three ready pods and sets the ReplicaSet's status.readyReplicas, the deployment controller sees that and marks the Deployment available. When you run kubectl rollout status, you are simply watching status fields that were populated bottom-up through the exact same API-server-and-watch machinery that carried the spec top-down. One store, one write path, both directions.

Part V: Internals deep dives

The reconcile loop, properly understood

The pattern deserves precision, because its adjectives carry the design. A controller is level-triggered: it acts on the current state (the level), not on the event that announced the change (the edge). Events are treated only as hints about when to look. The workqueue enforces this physically: informer handlers enqueue just a key like default/web, never the event payload, and the reconcile function re-reads the object fresh from cache and recomputes everything from scratch. Duplicate hints coalesce in the queue, and a missed hint costs nothing but latency, because periodic resyncs re-deliver every key anyway.

edge-triggered:  "pod X was deleted"  → handler must not miss it, ever
level-triggered: "something about default/web changed, go look"
                 → recompute: want 3, have 2 → create 1
                 (crash anywhere, restart, re-look: still correct)

This is why reconcile must be idempotent: it will run many times for one logical change, after crashes, after resyncs, after spurious wakeups, and running it twice must be harmless. And it is why controllers re-list on restart: a freshly started controller lists everything once, rebuilds its view of the level, and converges, with no replay log and no missed-event recovery protocol, because there is nothing to miss. The famous trap is writing an edge-triggered controller anyway, one that acts on event payloads ("on delete, do X"), which works in the demo and then silently corrupts state the first time a watch reconnects and events are compacted away. The correction is mechanical: reconcile reads only the key, and desired-vs-observed is recomputed every single time.

API machinery: group/version/kind and CRDs

Every object in the system is identified by a group/version/kind: our Deployment is apps/v1, Kind=Deployment; pods live in the original "core" group whose name is the empty string, which is why their URLs start with /api/v1 while everything else is under /apis/<group>/<version>. Versions are presentation, not storage: the server converts every version through an internal representation, so v1 and a hypothetical v2 of a type are two lenses on one stored object, which is how Kubernetes evolves APIs without migrations. The type system, schemes, and conversion machinery live in staging/src/k8s.io/apimachinery/, and the server publishes an OpenAPI schema of every type at /openapi/v3, which is what powers kubectl explain and client validation.

The payoff of all this indirection is that extension is first-class. A CustomResourceDefinition is an object that, when written, causes the API server (via the apiextensions-apiserver, staging/src/k8s.io/apiextensions-apiserver/) to start serving a brand-new REST resource, with the same storage, validation-by-schema, RBAC, and watch support as built-in types. Pair a CRD with a controller and you have an operator: the pattern of the entire cloud-native ecosystem, and the reason "extending Kubernetes" means adding nouns to the database rather than patching any binary. The misconception to correct: a CRD gives you an API and storage, and nothing else happens until you write the controller; the definition is inert data, the loop is the behavior.

Informers: why a thousand controllers do not crush the API server

If every reconcile re-fetched objects over the network, the API server would be the hottest path in the cluster. client-go's informer machinery, under staging/src/k8s.io/client-go/tools/cache/, is the load-bearing optimization, and its pipeline is worth memorizing:

API server
   │  one LIST, then one WATCH per resource type
   ▼
Reflector (reflector.go)          pulls the stream
   ▼
DeltaFIFO (delta_fifo.go)         orders the changes
   ▼
Indexer (store.go, thread_safe_store.go)   the local cache
   ▼
event handlers ──► workqueue ──► reconcile workers
                                  (which READ from the Indexer)

The reflector lists once, then watches from the returned resourceVersion, keeping the local cache current for the cost of one open stream. A SharedInformerFactory lets every controller in a process share one cache per type, so the controller manager, hosting dozens of controllers, holds one copy of the world. Reads during reconcile hit local memory and cost nothing. Two traps live here. First, the informer's "resync" does not contact the API server: it replays the local cache's contents into the handlers, a periodic level-trigger nudge; a true re-list happens only when a watch breaks and cannot resume (the "too old resource version" error, because etcd compacts history). Second, the cache is eventually consistent, so a controller may read a version of the world slightly older than its own last write; correct controllers tolerate stale reads, which is the level-triggered philosophy pushed into the client library, and sloppy ones double-create, which is exactly what the replicaset controller's expectations cache defends against.

The scheduler framework

Since the scheduling framework rewrite, the scheduler is a thin loop around ordered extension points, and every policy is a plugin implementing some of them:

scheduling cycle (serial, one pod at a time)
  PreEnqueue → QueueSort → PreFilter → Filter → PostFilter
             → PreScore → Score → NormalizeScore → Reserve → Permit
binding cycle (concurrent, per pod)
  PreBind → Bind → PostBind

Filter answers "can this pod run on this node" per node, and is where resources, taints, node affinity, and volume constraints live; PostFilter runs only when no node fits and is where preemption (evicting lower-priority pods) is implemented; Score ranks feasible nodes from 0 to 100 and weighted sums pick the winner; Reserve pessimistically accounts the pod's resources in the scheduler's cache before the (asynchronous) binding cycle confirms it, so the next pod in the serial scheduling cycle sees correct capacity. On large clusters the scheduler does not even score every feasible node; it samples a percentage, trading marginally better placement for throughput. The plugin registry is pkg/scheduler/framework/plugins/registry.go, and each subdirectory there is a small, testable answer to one placement question, which is why Part VI recommends them as an entry point. The trap to correct: the scheduler only ever places pending pods. It does not rebalance running pods when a better node appears or when you uncordon a node; if you want that, a separate component (the descheduler) must evict pods so they get scheduled again.

Part VI: Reading the repository

Honesty first: kubernetes/kubernetes is millions of lines of Go, one of the largest codebases in open source, and reading it front to back is how attempts die. The shape to hold in your head: cmd/ has the binaries' entry points, pkg/ their implementations, api/ the API specifications, and staging/src/k8s.io/ the code developed in-repo but published as separate libraries (client-go, apimachinery, apiserver, kubectl, cri-api). Then read it in stages, each of which is genuinely small.

Stage 0: sample-controller. Start outside the main repo with kubernetes/sample-controller, the official minimal controller: main.go and controller.go, a few hundred lines wiring an informer to a workqueue to a reconcile function for a toy CRD. After: you can draw the informer pipeline from memory and explain why handlers enqueue keys, not objects.

Stage 1: one real controller. pkg/controller/replicaset/replica_set.go, then skim pkg/controller/deployment/ (sync.go, rolling.go) to watch the same skeleton carry a harder policy. After: you can explain expectations, batch creation, ownerReferences, and how a rolling update is two ReplicaSets and a budget.

Stage 2: the informer machinery itself. staging/src/k8s.io/client-go/tools/cache/, in this order: listwatch.go, reflector.go, delta_fifo.go, shared_informer.go. After: you can say precisely what resync does, what happens on watch failure, and where a stale read can come from.

Stage 3: the scheduler. pkg/scheduler/schedule_one.go for the cycle, then three plugins: noderesources, tainttoleration, interpodaffinity under pkg/scheduler/framework/plugins/. After: you can name the extension points in order and explain Reserve's role in keeping the serial cycle correct beside concurrent binding.

Stage 4: the API server's storage path. pkg/registry/apps/deployment/strategy.go for one type's personality, then staging/src/k8s.io/apiserver/pkg/endpoints/handlers/create.go and storage/etcd3/store.go. After: you can trace a POST from HTTP handler to etcd transaction and explain resourceVersion end to end.

Stage 5 (optional): the kubelet. The hardest component, a decade of production hardening around one loop. Read pkg/kubelet/kubelet.go only for syncLoop, then pkg/kubelet/kuberuntime/ for the CRI translation. After: you can describe a pod start as CRI calls in order.

Where not to start: cmd/kube-apiserver/ and the generic apiserver's construction code, which is layers of configuration plumbing; anything named zz_generated*.go, which is machine-written; and apimachinery's conversion and serialization internals, which are best learned lazily, when a specific question forces you in.

Part VII: Hands-on labs

Lab 1: a cluster, and losing a fight with a controller. Teaches: reconcile beats imperative action. Create the default cluster and the web Deployment from Part II, then, with a watch running in a second terminal, delete a pod:

# terminal 1
kubectl get pods --watch
# terminal 2
kubectl delete pod $(kubectl get pods -l app=web -o jsonpath='{.items[0].metadata.name}')

Observe: the deleted pod goes Terminating and a replacement appears within roughly a second, before the old one is even gone. Nothing "restarted" your pod; the replicaset controller counted two, wanted three, and created. Then try kubectl delete rs -l app=web and watch the deployment controller recreate the entire ReplicaSet: the same loop, one level up.

Lab 2: watch a rolling update happen. Teaches: a rollout is two ReplicaSets and a budget (Part IV stage 6). With kubectl get pods --watch running:

kubectl set image deployment/web nginx=nginx:1.28
kubectl get rs      # run during and after

Observe: new-hash pods are created one or two at a time as old-hash pods terminate, never dropping below the availability budget; afterward kubectl get rs shows the old ReplicaSet retained at 0 replicas. Run kubectl rollout undo deployment/web and watch the zero-scaled ReplicaSet re-inflate: rollback is not a special mechanism, it is the same reconcile toward an older template.

Lab 3: a 50-line client-go watcher. Teaches: the watch stream from Part IV stage 5 is just an API you can consume.

mkdir podwatch && cd podwatch && go mod init podwatch
go get k8s.io/client-go@latest

package main

import (
    "context"
    "flag"
    "fmt"
    "path/filepath"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/util/homedir"
)

func main() {
    kubeconfig := flag.String("kubeconfig",
        filepath.Join(homedir.HomeDir(), ".kube", "config"), "")
    flag.Parse()

    config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
    if err != nil {
        panic(err)
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        panic(err)
    }

    w, err := clientset.CoreV1().Pods("default").
        Watch(context.Background(), metav1.ListOptions{})
    if err != nil {
        panic(err)
    }
    defer w.Stop()

    for event := range w.ResultChan() {
        pod, ok := event.Object.(*corev1.Pod)
        if !ok {
            continue
        }
        fmt.Printf("%-9s %-42s phase=%-9s node=%s\n",
            event.Type, pod.Name, pod.Status.Phase, pod.Spec.NodeName)
    }
}

Run it, then delete a pod in another terminal. Observe: the deletion arrives as several MODIFIED events (the deletion timestamp, status updates) before the final DELETED, and the replacement's life arrives as ADDED with an empty node, then MODIFIED with a node name (the scheduler's bind), then MODIFIED to Running (the kubelet's status write). You are watching Part IV's stages 5 through 10 as raw events. A raw watch like this drops events on reconnect; production code uses an informer, which is this plus the cache and re-list machinery from Part V.

Lab 4: read etcd directly. Teaches: etcd is the only state there is. kind's control plane runs etcd as a static pod that ships etcdctl, so exec into it, presenting etcd's client certificates:

kubectl exec -n kube-system etcd-kind-control-plane -- sh -c \
  "ETCDCTL_API=3 etcdctl \
     --cacert /etc/kubernetes/pki/etcd/ca.crt \
     --cert   /etc/kubernetes/pki/etcd/server.crt \
     --key    /etc/kubernetes/pki/etcd/server.key \
     get /registry/deployments/default/web --prefix --keys-only"

Observe: the key exists; drop --keys-only and the value is mostly unreadable, because objects are stored as protobuf. List /registry/pods/default --prefix --keys-only and see one key per pod. Now delete a pod and immediately re-list: the old key vanishes and a new one appears, confirming that every step of lab 1 was, physically, key writes in this store.

Lab 5: drain a node. Teaches: cordon/evict semantics, and that the scheduler never moves running pods. This needs a multi-node cluster; put this in kind-3node.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

kind create cluster --name lab --config kind-3node.yaml
kubectl apply -f web.yaml
kubectl scale deployment web --replicas=6
kubectl get pods -o wide          # pods spread across lab-worker, lab-worker2
kubectl drain lab-worker --ignore-daemonsets
kubectl get pods -o wide          # all six now on lab-worker2
kubectl uncordon lab-worker
kubectl get pods -o wide          # still all on lab-worker2

Observe: drain cordons the node (marks it unschedulable) and evicts its pods; the controllers replace them, and the scheduler, filtering out the cordoned node, lands every replacement on the other worker. The final step is the real lesson: after uncordon, nothing rebalances, because scheduling is a one-time decision for pending pods (Part V). Delete a pod and its replacement may now land on lab-worker. Clean up with kind delete cluster --name lab.

Lab 6: watch the control plane converge after a partition. Teaches: level-triggering survives missed events. Stop the controller manager by moving its static pod manifest aside inside the kind node, delete a pod (nothing replaces it, and the Deployment does not notice), then restore the manifest:

docker exec kind-control-plane sh -c \
  "mv /etc/kubernetes/manifests/kube-controller-manager.yaml /root/"
kubectl delete pod -l app=web --wait=false   # pods die, none return
kubectl get pods                             # fewer than 3, indefinitely
docker exec kind-control-plane sh -c \
  "mv /root/kube-controller-manager.yaml /etc/kubernetes/manifests/"
kubectl get pods --watch                     # replacements appear

Observe: when the controller manager returns, it does not need a log of what it missed. It lists current state, compares against desired state, and creates the missing pods. That is level-triggered reconciliation demonstrated with a hole blown in the event stream.

Part VIII: Questions and model answers

Q1. What actually happens, synchronously, when kubectl apply creates a Deployment?

Only stages 1 to 4: the request is authenticated, authorized, admitted, validated against the Deployment strategy, and written to etcd, and the server returns 201. No ReplicaSet, no pods, no containers exist yet; everything else is asynchronous controllers reacting to the watch event. "Created" means "recorded."

Q2. Why is level-triggered reconciliation the right choice over edge-triggered event handling?

Because correctness stops depending on delivery. An edge-triggered system must never miss an event, so crashes, reconnects, and etcd history compaction all become correctness hazards requiring replay protocols. A level-triggered controller recomputes desired-vs-observed from current state every pass, so a missed event costs only latency, and restart recovery is just "list and reconcile," the same code as normal operation.

Q3. How do components coordinate if none of them talk to each other?

Through shared state: every component reads and writes objects via the API server and reacts to changes via watches. The deployment controller does not call the replicaset controller; it writes a ReplicaSet object, and the replicaset controller notices. This hub-and-spoke design centralizes security and validation, lets components restart independently, and makes extension a matter of adding object types.

Q4. What is resourceVersion and how does Kubernetes handle concurrent writes?

resourceVersion is the object's etcd modification revision. Updates are optimistically concurrent: a write must carry the version it read, and if the object changed meanwhile, the API server returns 409 Conflict and the client re-reads and retries. There are no distributed locks anywhere in the write path.

Q5. Walk through what an informer does and why it exists.

An informer lists a resource once, then watches from that resourceVersion, maintaining a local in-memory cache (the Indexer) and delivering events to handlers, which typically enqueue object keys into a workqueue drained by reconcile workers reading from the cache. It exists so that thousands of controllers can each do unlimited reads at memory cost, with the API server serving one watch stream per informer instead of a query per read.

Q6. What does informer "resync" actually do?

It replays the local cache's contents into the event handlers on a timer; it does not contact the API server. It is a periodic level-trigger nudge that re-delivers every key so reconcile gets another chance at anything stuck. A true re-list against the server happens only when a watch breaks and cannot resume from its resourceVersion because etcd has compacted that history.

Q7. Why must reconcile functions be idempotent, and how is that typically achieved?

Because one logical change triggers many reconcile runs: coalesced queue hints, resyncs, restarts. The standard techniques are recomputing everything from current state rather than applying deltas, checking before creating (or using deterministic names so duplicate creates fail), and tracking in-flight actions, as the replicaset controller's expectations cache does, so the loop does not act again before its own effects are visible.

Q8. What are the scheduler framework's main extension points and what runs where?

The scheduling cycle runs serially per pod: PreFilter and Filter eliminate infeasible nodes, PostFilter handles the no-fit case (preemption), PreScore/Score/NormalizeScore rank survivors, and Reserve accounts resources in the scheduler cache before Permit. The binding cycle (PreBind, Bind, PostBind) runs concurrently and performs the actual API write to the pod's binding subresource. Reserve is what keeps the serial cycle correct while binds are still in flight.

Q9. Why does the scheduler never touch a node, and why does it never move a running pod?

Its entire output is one field, spec.nodeName, written through the API; the kubelet does all node-local work. And it only considers pending pods, so placement is a one-time decision: uncordoning a node or adding capacity rebalances nothing until pods are recreated, which is why the descheduler exists as a separate evict-to-reschedule component.

Q10. What is the CRI and why does it exist?

The Container Runtime Interface is the gRPC API (RuntimeService and ImageService, defined in staging/src/k8s.io/cri-api/) between the kubelet and container runtimes like containerd and CRI-O. It exists so the kubelet is runtime-agnostic; Docker-specific integration was removed once the interface fully decoupled Kubernetes from any one runtime.

Q11. What is stored in etcd, in what format, and who may access it?

Every API object, serialized as protobuf under keys like /registry/pods/<namespace>/<name>. Only the API server talks to etcd; all other components go through the API. That discipline is what makes authn, authz, admission, and validation universally enforceable, and it is why etcd access is equivalent to root on the cluster.

Q12. How does a rolling update work mechanically, and what is a rollback?

The deployment controller keeps one ReplicaSet per pod-template hash. On a template change it creates a new ReplicaSet and shifts replica counts between old and new within the maxSurge/maxUnavailable budget, driven by status flowing back up as pods become ready. Old ReplicaSets are retained at zero replicas as revision history, so rollback is just scaling an old ReplicaSet back up via the same reconcile.

Q13. When would you recommend against Kubernetes?

A single application with a small team is better served by a VM with Docker Compose, or by serverless container platforms like Cloud Run or Fargate that remove cluster operations entirely. Nomad suits teams that want scheduling without the API machinery; ECS suits AWS-only shops. Kubernetes pays off with many services, many teams, and the need for its extensible API, and not before.

Q14. A pod is stuck Pending. How do you debug it?

kubectl describe pod and read the Events: a scheduling failure names the filter that rejected each node (insufficient cpu/memory, untolerated taints, affinity or volume constraints). If there is no scheduling event at all, check that the scheduler is running and the pod has no schedulingGates. If it is bound but not starting, the problem has moved to the kubelet: image pulls, volume mounts, or the runtime, all visible in the same events stream.

Q15. Why is writing status into the same object as spec, through the same API, a good design?

It gives one consistent channel for both directions: desired state flows down and observations flow up through identical storage, watch, and access-control machinery, so any client can observe convergence by watching status fields. Separating status as a subresource keeps ownership clean: users write spec, controllers write status, and RBAC can enforce exactly that.

Q16. What breaks if an external system also manages "what should be running"?

You get two sources of truth, and the controllers, which treat the API server's state as authoritative, revert the external system's out-of-band changes on every reconcile. The system appears haunted and drifts forever. The fix is to make the external system write desired state into the API (the GitOps pattern) so there is exactly one authority and the loops work for you instead of against you.

Part IX: Design lessons

1. Make the store the interface. Components coordinate through one consistent database instead of calling each other, which decouples their lifecycles and centralizes policy. You see the same move in event-sourced architectures and in the blackboard pattern; the systems design write-ups reach for it whenever "N components must agree" threatens to become N² protocols.

2. Level-triggered beats edge-triggered when you can afford to re-look. Recomputing from current state turns missed events, crashes, and races from correctness bugs into latency. The same principle underlies retry-until-converged configuration management (Terraform plan/apply, Puppet runs) and React's render model: describe the target, re-derive from scratch, diff.

3. Optimistic concurrency scales where locks do not. A version number and a 409 replaced every distributed lock in this system. The identical mechanism appears as compare-and-swap in CPUs, conditional writes in object stores and DynamoDB, and ETags in HTTP.

4. Extend by adding nouns, not by patching verbs. CRDs let third parties add resource types that inherit storage, validation, RBAC, and watches for free, which is why the operator ecosystem could grow without forking the core. Compare Postgres extensions and LSP: a stable, generic core protocol plus first-class user-defined types is how platforms outlive their authors' imaginations.

5. Spend complexity where it buys leverage, and make the leaf nodes boring. The cleverness concentrates in the API machinery, informers, and scheduler; an individual controller is a readable count-and-converge loop, and that boringness is what lets thousands of people write correct ones. vLLM makes the same trade in the opposite domain, keeping model code plain while memory and scheduling carry the cleverness; it is a hallmark of infrastructure that intends to be extended.

Part X: Memorization framework

The one-sentence summary: Kubernetes is a consistent database of desired state behind one API server, converged into reality by independent level-triggered reconcile loops. The chain to hold:

apply → authn → authz → admit → validate → etcd
      → watch → deploy-ctrl → RS-ctrl → sched(filter→score→bind)
      → kubelet → CRI → containerd → status flows back ↑

Mapped to source:

apply .......... staging/src/k8s.io/kubectl/
gauntlet ....... staging/src/k8s.io/apiserver/ + plugin/pkg/admission/
validate ....... pkg/registry/apps/deployment/strategy.go
etcd write ..... staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go
deploy-ctrl .... pkg/controller/deployment/sync.go, rolling.go
RS-ctrl ........ pkg/controller/replicaset/replica_set.go
scheduler ...... pkg/scheduler/schedule_one.go + framework/plugins/
kubelet ........ pkg/kubelet/kubelet.go (syncLoop) + kuberuntime/
CRI ............ staging/src/k8s.io/cri-api/
informers ...... staging/src/k8s.io/client-go/tools/cache/

Memorize these:

• The controller shape: ListWatch → Reflector → DeltaFIFO → Indexer → handlers → workqueue (keys only) → reconcile reading from cache. Every controller, everywhere.
• The scheduler pipeline: PreFilter → Filter → PostFilter → Score → Reserve → Permit, then PreBind → Bind → PostBind; scheduling serial, binding concurrent, pending pods only.
• Three invariants: only the API server touches etcd; controllers never call each other; users write spec, controllers write status.
• Two corrected traps: resync replays the cache, it does not re-list; a CRD is inert data until a controller gives it behavior.