[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8},"actionData","errors","root",{"_2212":2213},"routes/blog.author.$slug",{"_9":10,"_11":12,"_13":14,"_5":15},"author",{"_25":53,"_54":55,"_56":53},"blogs",[24],"pagination",{"_16":17,"_18":19,"_20":19,"_21":17,"_22":23},"https://planetscale.com","page",1,"hasPrev",false,"hasNext","totalPages","url","https://planetscale.com/blog/author/fatih",{"_25":26,"_27":28,"_29":30,"_31":-7,"_32":33,"_34":35,"_36":37,"_38":39,"_40":41,"_42":43,"_44":45},"slug","the-feedback-loops-behind-kubernetes","body",[138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298],"body_text","For the last decade, Kubernetes has been the backdrop to most of my work: operating clusters, helping build hosted Kubernetes, and writing Kubernetes operators. At PlanetScale, that now means running stateful systems like Postgres and MySQL in production. Kubernetes has many faces, but here I want to talk about one face only: why it is so good at running workloads at scale.\nPeople ask me what an operator actually does. The canonical answer is: \"it reconciles desired state.\" This is correct, but it also tells you almost nothing.\nAn operator is a feedback controller. It's the same closed loop that runs a thermostat or keeps your car at a fixed speed on cruise control. In our case, the thing being controlled is a database. I have been building these loops for years, and the best way I know to make them click is to ignore Kubernetes at the beginning. Kubernetes is full of control theory, even if we don't call it that in the day-to-day.\nBefore we look at a single line of Kubernetes, we're going to run a production database by hand and slowly let the feedback loop appear on its own. Then we'll map that loop to Kubernetes, with the pieces production needs: a store, watches, queues, retries, and more. At the end, we'll look at what one of these loops looks like in a real operator.\nA working understanding of containers and kubectl helps, but you don't need to be a Kubernetes expert. I'll use terms like idempotent, fan-in, and eventual consistency, and introduce the parts that matter as we go.\nWe're going to start slow and gradually ramp things up. Each part builds on the previous.\nPart 1: running Postgres by hand\nOne container, one machine\nLet's start from scratch. I want to run Postgres on a Linux box, and I need it inside a container. To start it, we run:docker run -d --name pg \\\n -e POSTGRES_PASSWORD=secret \\\n postgres:18\n\nThat's it. Postgres is running. My app connects to it, writes some rows, and everything works fine. But then the machine goes away: the cloud provider reclaims the instance (hardware fails, or a spot instance gets taken back), or I ship a new version of my setup, which means stopping the old container and starting a fresh one in its place. Either way, the container is replaced, and my data is gone. The container storage was ephemeral, and I did not attach any persistent volume to it.\nThere is already a gap between what I want (Postgres, running, with my data) and what I have (a container whose storage disappears when the container or node goes away). The rest of this post is about that gap and the machinery we build to close it.\nPick a node, by hand\nImagine we have hundreds of nodes (servers) we can use. I already have other workloads running on them. I need to decide which one runs this database. So I ssh into the box that looks the least busy and start the container there.ssh node-07 'docker run -d --name pg ... postgres:18'\n\nI picked node-07 because it looked idle enough. I start keeping track of it, save it in some sort of config file, and push it to some repo.\nIt needs a real disk\nContainer storage is ephemeral, so I have to attach a real block device. In the cloud this is an EBS volume (e.g. on AWS); on bare metal it's a physical disk. Assuming it's a block device, this is what we usually do: provision the volume, attach it to the node, format it, mount it, and point Postgres' data directory at the mount.# provision + attach first with cloud CLI, then on the node:\nmkfs.ext4 /dev/nvme1n1\nmkdir -p /var/lib/pg-data\nmount /dev/nvme1n1 /var/lib/pg-data\ndocker run -d --name pg \\\n -e POSTGRES_PASSWORD=secret \\\n -v /var/lib/pg-data:/var/lib/postgresql \\\n postgres:18\n\nThese are a lot of steps, and each one can fail halfway. And if the disk fills up later, Postgres stops accepting writes and we have to resize the volume by hand: first through the cloud provider, then again inside the filesystem.\nOne isn't enough\nA single Postgres instance is a single point of failure. We want high availability: one primary and two replicas. These need to be on three different machines, with streaming replication between them. So we do the same steps again, three times, on node-07, node-12, and node-19. I also wire up replication by hand: primary_conninfo, replication slots, all of it.\nNow we have three nodes with three Postgres instances. One of the instances is the primary (here it's node-07). But this raises new problems, like what to do if the primary's node dies?\nThey have to find each other\nHere is another thing we have to solve. The replicas need to reach the primary, and the primary needs to accept their connections. And every one of these addresses is an IP that changes when a container restarts.\nThe first thing I do is hard-code the IPs. I write node-07's address into the replicas' config, I list the replicas' addresses in the primary's pg_hba.conf, and I keep a small /etc/hosts table and save it somewhere.# on each replica's postgresql.auto.conf, until the primary is recreated with a new IP\nprimary_conninfo = 'host=10.4.7.21 port=5432 user=replicator ...'\n\nBut we still have a problem: the first time the primary is recreated with a different IP, the whole cluster falls apart.\n\nThe watchdog script\nNow, this is where we start thinking about how to solve these issues. Everything described so far can break, and will continue to break even if I fix it:\nA replica process dies and doesn't come back.\nA disk gets full.\nThe primary fails and a replica has to be promoted.\nA config I changed on two nodes but forgot on the third one. They are now out of sync.\nLet's assume we've set up a simple uptime monitor and we're going to get paged for all these cases. To avoid getting paged at night, we do the sensible thing: write a script. So we decide to write a loop that wakes up every few seconds, looks at each node, and fixes whatever's wrong.while true; do\n for node in node-07 node-12 node-19; do\n if ! ssh \"$node\" 'pg_isready -q'; then\n ssh \"$node\" 'docker start pg' # it died, bring it back\n fi\n\n usage=$(ssh \"$node\" \"df --output=pcent /var/lib/pg-data | tail -1 | tr -dc 0-9\")\n if [ \"$usage\" -gt 80 ]; then\n grow_volume \"$node\" # disk filling, make it bigger\n fi\n done\n sleep 5\ndone\n\nIt's written in Bash, and probably has tons of bugs. You notice something here? The loop doesn't care how the database got into a bad state. Every five seconds it looks at the current state of the world and asks this question: does reality match what I want?\nIf a process is down, start it. If a disk is filling, grow it. Run the loop once or run it a thousand times and the result is the same, because each action is conditional on the current state. The script is idempotent.\nChanging a parameter\nLet's make things a little more complex. I need to raise max_connections from 100 to 500. This one is not a reload-only change. PostgreSQL says it can only be set at server start, so the manual version is to ssh into each box, edit postgresql.conf, restart Postgres, and check that it took on all three.\nBecause I know that ssh'ing into the nodes manually isn't a thing I want anymore, I do the same thing we did previously: I write the desired value down in one place and teach the loop to enforce it.WANT_MAX_CONNECTIONS=500\n\nfor node in node-07 node-12 node-19; do\n have=$(ssh \"$node\" \"psql -tAc 'show max_connections'\")\n if [ \"$have\" != \"$WANT_MAX_CONNECTIONS\" ]; then\n ssh \"$node\" \"sed -i 's/^max_connections.*/max_connections = $WANT_MAX_CONNECTIONS/' /var/lib/pg-data/postgresql.conf\"\n ssh \"$node\" \"docker restart pg\"\n fi\ndone\n\nThis is the same idea as before. I read what I want (a variable). Observe what I have (a query). If they differ, I take an action to close the difference. Again, I don't track whether I changed it last time. All I do is compare and converge, every loop.\nWhat we actually built\nI started with a desired state that was written down in one place: three instances, this disk size, max_connections = 500. Every few seconds I observe the actual state of the system. I compute the difference. I take whatever action closes that difference. Then I do it again, forever.\nThat's a closed feedback loop. The word \"closed\" matters. It means the output of the system is fed back into the next decision. I don't run docker start and assume the database is fine. I check the database again. If it is still wrong, I act again. If it is already correct, I do nothing.\nThe nice part is that the same loop works for different problems. It can restart a dead process, grow a disk, or push max_connections = 500. The action changes, but the shape stays the same: read what I want, observe what I have, compare them, act, repeat. If I draw the same thing as a block diagram, with the control theory names added, it would look like this:\n\nHere is how the vocabulary from control theory maps cleanly onto my shell script:\nThe setpoint is my desired state, the variables at the top of the script (disk size, max_connections and so on).\nThe measured output is what I observe: pg_isready, df, show max_connections.\nThe error (e) is the difference between them.\nThe controller is the body of the loop, the if statements that decide what to do. It is not the whole script.\nThe actuator is what carries out the action: ssh plus docker start.\nThe plant is the system being controlled, Postgres and its disk.\nThat also gives us a nice way to understand open-loop control. My very first attempt, ssh in, run the command, and walk away, was open-loop: fire an action and assume it worked. The Bash script is closed-loop because it keeps feeding the measured state back into the next decision.\nA Bash loop is not a production control plane. Just to name a few issues with it:\nIt has no concurrency control, so two copies of the script can race each other. Imagine both deciding to promote a different replica.\nIt keeps its only real state, \"am I mid-failover?\", in a shell variable that could die with the process.\nIt polls every node every five seconds whether anything changed or not, which is fine for three nodes, but too expensive for three thousand nodes.\nIt has no idea what to do when the ssh itself times out.\nAnd the moment I want a second kind of resource, a connection pooler, a backup job, a read replica in another region, I'm copy-pasting this whole structure.\nWhat if the script also fails? Who runs it then? We could keep hardening this script, but look at where it goes: we would need a real store for the desired state, watches instead of polling, a work queue, retries, leader election. We would be rebuilding Kubernetes. The real platform already exists, and it's Kubernetes.\nPart 2: how we reinvented Kubernetes\nNow we can map what we hand-rolled in Part 1 to Kubernetes. Almost all of it already exists there. The operator is the part we care about.\nThe other loops\nLet's go through some of the pieces we built by hand before the watchdog loop. You already know these components by name. What you might not have noticed is that they also work like controllers.\nSpinning up the container: the kubelet. First, a quick definition: a Pod is the smallest thing Kubernetes runs, one or more containers scheduled together on a node and sharing its network. For us it's the Postgres container. On every node runs an agent called the kubelet. Its desired state is the set of Pods assigned to its node, which it learns from the API server. Its observed state is the set of containers actually running, which it gets from the container runtime. When they differ, it starts the missing container, kills the extra one, or restarts the crashed one. My if ! pg_isready; then docker start; fi is the kubelet's job, just done properly. The kubelet doesn't shell into anything; it talks to containerd over a gRPC socket, which talks to runc.\nPicking a node: the scheduler. Remember me choosing node-07? That's the scheduler's whole reason to exist. It watches for Pods with no node assigned, filters out the nodes that can't work, scores the rest, and writes the decision to one field: pod.Spec.NodeName. The scheduler doesn't start the container; it records the placement and lets the kubelet pick it up. You will realize that most things in Kubernetes are decoupled like this.\nAttaching the disk: CSI and the PV/PVC sync. My multi-step mkfs and mount script becomes a PersistentVolumeClaim, which is a declarative request for storage. The Container Storage Interface (CSI) driver turns that request into a real volume. CSI itself is a set of controllers and sidecars: one provisions, one attaches, one resizes, and so on, while the kubelet calls the driver's node plugin to do the actual mount. It's a family of controllers. If there is a PVC but no disk behind it, one controller creates the disk. If the PVC size increases, another controller calls the provider API (e.g., AWS ModifyVolume). Again, I write intent, and a controller does the actual work. (note: I wrote one of the early production CSI drivers, csi-digitalocean, and a long post about building one.)\nMaking them find each other: the CNI and Services. The /etc/hosts problem is solved at a layer we no longer have to think about. A CNI plugin gives Pods their network identity; Cilium, for example, does this with eBPF instead of a pile of iptables rules. For stateful workloads, a StatefulSet plus a headless Service gives each replica its own stable DNS name, which is exactly what a Postgres replica needs. The hard-coded IP that broke our cluster becomes a name that keeps working. DNS is only one tool here; other service discovery systems like etcd, ZooKeeper, and Consul solve similar problems.\nAs you see, all the problems we solved with ssh and various scripts are replaced by Kubernetes components and drivers. And these are just a few of them:\n\nAll of this so far is useful context, but the part we care about is our watchdog loop, because that's the one we get to write ourselves.\nThat's the operator.\nThe for-loop translated to Kubernetes\nIn Kubernetes, our watchdog script is a controller, and the standard way to write one in Go is a library called controller-runtime. At its heart, it's a function with a basic signature:func (r *Reconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {\n // req contains a namespace/name. That's it. That's the whole input.\n}\n\nNotice what is missing here: the function isn't told what changed. There is no diff. It isn't handed the old object and the new object. It isn't given an event type. It gets a key, a namespace and a name, and nothing else. It's minimal by design, because it has to work for many different controllers. The function's job is to fetch the object with that namespace/name, look at the world, and converge to the desired state.\nEdge-triggered notifications, level-triggered logic\nThere are two ways to build any closed feedback loop:\nEdge-triggered: act on transitions, on events. \"The disk crossed 80%.\" \"The Pod was deleted.\" \"The number of replicas increased by 2.\"\nLevel-triggered: act on the current state, regardless of how you got there. \"The disk is at 85%.\" \"The Pod is missing.\" \"The number of replicas is 3.\"\nMy first mental model of controllers, and probably yours at some point, was edge-triggered: listen to a stream of changes, and for each change, try to converge.\nThe problem is that this is very fragile. In distributed systems, if one component is fragile, the fragility spreads to the rest of the system. Why is edge-triggering fragile? Say your controller is down for thirty seconds. It misses the events from those thirty seconds, and its view of the world is now permanently wrong. If two events arrive out of order, you process them out of order. If an event is delivered twice, you act twice. You're rebuilding your state from a stream of events, and you've inherited all of event sourcing's hard problems.\nHere is a very concrete example. Assume you have 1 replica, and you increase it to 3 replicas. Because you have only subscribed to changes, either:\nYou miss the event (maybe the queue dropped it, or the consumer, your app, dropped it due to a crash or a full buffer).\nYou receive it twice.\nIn the first case, you won't be able to self-correct. In the second case, if your handler blindly applies the delta again, you'll end up with 5 replicas (you overshoot), instead of 3.\nThe level-triggered model fixes all of that. Remember, our shell script never asked \"what changed?\" It asked \"what is true right now?\", every five seconds, from scratch. Miss a loop, and the next one catches up. Run the loop twice, and you get the same result. The current state of the world is the only input that matters, and it's always available to read. So in the level-triggered case, our example above becomes this: you read replicas=3, you check the current number of replicas, which is 1, and you increase by 2.\nIf you miss the event, no one cares. In the next reconcile loop you'll catch it. If your app crashes, it comes back, reads again and detects that it did not increase it yet, increases it.\nKubernetes controllers combine both: edge-triggered notifications, level-triggered logic.\nEvents (the edges) are only a hint that it's worth looking again. They tell you when to reconcile, never what to do. The reconcile itself is level-based: it reads the current state (e.g., replicas=3) and drives toward the desired state (e.g., create 2 replicas), ignoring the triggering event completely. That's why Reconcile only gets a key. The framework makes it hard to write edge-triggered logic, on purpose. Edge-triggered logic is how you get a controller that's fragile and permanently wrong after its first hiccup.\n\nOur bash script stumbled into this property by accident, at least for the sake of the example. But the controller-runtime framework gives it to you on purpose. It's why a Kubernetes controller can crash, get restarted ten minutes later, and converge correctly with no special recovery code. There is no recovery code. There is just the loop. The controller can reconstruct the world from scratch.\nInformers, the work queue, and a cache\nSo where do the edges come from? And what stops a controller from DDoSing the API server by listing everything every five seconds like my script did?\nThe answer is the informer. An informer opens a single watch against the API server for a given resource type, streams every add, update, and delete, and keeps a complete in-memory cache of the objects we're interested in. Two things matter here:\nFirst, the informer turns each watch event into a key and puts it on a work queue. The queue does a lot of work for you.\nIt coalesces: if the same object is updated five times before you get to it, you reconcile it once, against the latest state (level-triggered again).\nIt rate-limits: an object that keeps erroring backs off exponentially instead of spinning. This is damping, the same reason a crash-looping container backs off instead of restarting hot.\nIt lets you run a pool of workers pulling keys in parallel, which is your fan-out. Events fan in from the watch, collapse in the queue, and fan out to the workers. This is something you need to tune. The higher you set the pool, the more pressure you put on the system: more writes, more API calls, more load on the provider, and more CPU usage in the operator.\n\nSecond, and this is a detail that bites people a lot: your reads and your writes in Kubernetes don't go to the same place.\nIn controller-runtime, the client you're handed reads from the informer's local cache. Cache reads are cheap, they don't touch the API server, and that's how a controller reconciles thousands of objects without falling over. But your writes go straight to the API server. The cache only learns about your write when the resulting watch event comes back around, a moment later.\nBecause of that, a read can be stale. You need to be prepared for this.\nIf you write a field of an object and then read the same object again from the cache, the reconciler might think it's not updated yet. You write again, and you get a Conflict error. Retrying with a fresh read can be fine, but blindly retrying against the same stale cached view just spins.\nMost of the time, what you want is to drop the call and requeue. In the next reconcile, the GET will see the updated object, and your write will never happen. That's how everything self-converges.\nHere is another edge case. Picture this sequence inside a reconcile:// I want N replicas. I see fewer, so I create the missing ones.\nexisting, _ := r.listChildPods(ctx) // reads the CACHE\nfor i := len(existing); i < desired; i++ {\n r.client.Create(ctx, newPod(i)) // writes the API SERVER\n}\n\nNow an event fires again a second later, before the cache has caught up with the Pods you just created. You list from the cache, and the new Pods aren't there yet. Your code decides it still needs to create them, and you create duplicates. This is the classic stale-cache double-create, and it's nasty because it only shows up under timing you can't reproduce on your laptop.\n\nThere are two ways out. The correct one is the expectations pattern, the same trick the built-in ReplicaSet controller uses: you record that you expect to see N creations in memory, and you don't act again until the cache has caught up to your own writes. It works, but it's not easy to implement and it's a fair amount of machinery. Read more on Ahmet's blog.\nThe pragmatic one, which a lot of people use, is to bypass the cache for the reads where a stale view would cause a double-create or double-delete, and go straight to the API server:// The cached client can be stale right after our own writes, which\n// would make us miscount and create duplicates. For this one read,\n// go direct to the API server instead of the cache. Slower,\n// but consistent for this decision.\nerr := r.apiReader.List(ctx, &instances, client.InNamespace(ns), labelSelector)\n\nThis is not only a Kubernetes issue. In any system with a read cache and a write-through path, read-after-write is not consistent unless you make it so. Most of the time the cache is exactly what you want: cheap, local, and eventually consistent. Eventual consistency is fine because the loop runs again. But the moment a decision would be destructive or non-idempotent if you acted on a stale read, you need to know which path you're on. Kubernetes solves many hard problems, but it also gives you a few new ones.\nSetpoint and measured variable: spec and status\nBack to the control diagram. My script kept its setpoint in shell variables and its measured state in the output of df and psql. Kubernetes gives both a permanent home, on the object itself.\n.spec is the setpoint, the desired state. It's owned by whoever created the object (a human, or another controller), and the reconciler treats it as read-only intent. It's an anti-pattern to write to the .spec from inside the controller. If you do it, stop reading, go and fix your codebase. There are only a handful of exceptions, but a controller should generally never set its own setpoint.\n.status is the measured variable, the observed state. It's owned by the controller, written through a separate status subresource, and it's where you record what's actually true. The better the status, the better the controller can decide. A good .status field is what makes a controller pleasant to operate. The word observability comes from control theory; Kalman coined it around 1960 to ask whether you can infer a system's internal state from its outputs. .status is also your response to any third-party system. If someone wants to learn the outcome of your actions, .status is the place to look at.\nThat split is the whole declarative model in two fields. It comes with a piece of bookkeeping that's pure control theory: .metadata.generation increments when desired state changes, and by convention the controller writes back .status.observedGeneration to say \"the state I'm reporting reflects this version of your intent.\"\nWhen observedGeneration < generation, the status you're looking at does not reflect the latest setpoint yet. That one comparison is how you tell \"converged\" from \"still working on it.\"\n\nThis is why the reconcile is stateless, and why that matters. Our shell script kept \"am I mid-failover?\" in a variable that died with the process. A Kubernetes controller keeps nothing important in memory. Every fact it needs is on an API object: the spec it's driving toward, the status it last observed, the conditions describing where things stand. Kill the controller, restart it on another node, and it picks up exactly where it left off, not because it saved its progress, but because there was never any in-memory progress to lose. The state lives in the cluster (API server, etcd is what holds the state). The controller is just the loop that reads it.\nSelf-healing by design\nThis is the part I like most.\nWhen a controller creates a child object (a Pod, a PVC), it stamps an ownerReference on the child pointing back at the parent. That reference does two things. It sets up garbage collection: delete the parent, and Kubernetes can cascade the delete to its children. And it gives the controller a way to map child changes back to the parent: \"when any object I own changes, enqueue my parent for a reconcile.\" ownerReference allows you to link controllers to each other and create chains. If done right, all your controllers and systems fit together.\nHere is an example. Follow the loop:\nA node dies and takes a Pod with it.\nThe Pod's deletion is a watch event, an edge.\nThrough the ownership link, that edge becomes a reconcile request for the parent.\nThe parent reconciles, observes its children (level-triggered), sees one is missing and the count is below the setpoint, and creates a replacement.\nThe replacement is an unscheduled Pod, an edge for the scheduler.\nThe scheduler detects the unscheduled Pod, assigns a node.\nThe kubelet gets triggered because that's an edge for that node's kubelet and it starts the container.\nThat's multiple feedback loops, each watching the layer below, each reacting to an edge and converging to its own level, chained together through the API server with nobody orchestrating the whole thing.\nControl theory has a name for loops stacked like this: cascade control. The output of an outer loop becomes the setpoint of an inner loop. A controller never writes its own .spec, but it writes other objects' .spec all the time. My operator writes the PVC's spec, and that spec is the setpoint the CSI controllers converge to. Each loop worries only about its own layer and trusts the loop below.\nSo we wrote if ! pg_isready; then docker start; fi and maybe thought we're good. Kubernetes turns that one line into several independent controllers that have never heard of each other, but still cooperate because they share the API server and watch each other's objects. I like this part a lot. Nobody calls a central orchestrator. Nobody passes a private message. The system heals itself.\nWhat \"observe\" actually means in a real operator\nUp to here I've been a little vague about the \"measure\" step, because in the examples the measured state is just \"list the child Pods.\" But in a real database operator it's a lot more than that.\nWhen an operator I work on reconciles a single Postgres instance, the first thing it does, before it decides anything, is build a snapshot of reality from every source that knows something true about that instance. Not just Kubernetes. Kubernetes barely knows anything about whether Postgres is actually healthy.\nThe sources gathered at the top of every reconcile:\nThe Kubernetes cache: the Pod, its PVC, the PV behind it, the Node it's on, the ConfigMap holding its config. These are the cheap local reads, the stuff we already talked about.\nThe database's effective configuration. Not what we last wrote down, but what the server has actually loaded, so we can compare the two and detect drift. Other entities can rewrite or reload the config on disk without us knowing, so the only honest source of truth is the running server itself, never our last write.\nThe database's own view of its health. Its role, whether it's healthy, how far behind its followers are, whether it's currently accepting writes. Some of this comes from the agents that sit next to the database and manage it; some we get by opening a connection and asking the database directly. These calls carry a tight timeout and are allowed to fail, more on that below.\nA background collector. Some signals are too expensive or too rate-limited to fetch on every reconcile: disk usage, or whether a volume operation we kicked off earlier is still in flight and where it sits in its cooldown window. A separate collector, often a background goroutine, gathers these on a slow cadence and keeps the last value per volume in memory. The reconcile reads that value instantly, without blocking on anything. Think of these as custom workqueues you implement.\n\nIn code, the snapshot is just a struct, and the reconcile's first move is to populate it. This is simplified, but faithful to the real shape:// The observation snapshot: everything we know about this instance, right now.\ntype reconcileHandler struct {\n // The object (.spec = setpoint, .status = measured).\n instance *v1.PostgresInstance\n\n // Kubernetes objects.\n pod *corev1.Pod\n pvc *corev1.PersistentVolumeClaim\n node *corev1.Node\n\n // Database state.\n dbState DatabaseState\n\n // What Postgres actually loaded, not what we last wrote.\n effectiveConfig map[string]string\n\n // Collected out-of-band.\n diskUsage *resource.Quantity\n\n // The volume operation already in flight, if any.\n storageOp *StorageOperation\n}\n\nfunc (r *Reconciler) newReconcileHandler(\n ctx context.Context,\n inst *v1.PostgresInstance,\n) (*reconcileHandler, error) {\n h := &reconcileHandler{instance: inst}\n\n // Cheap local reads.\n h.pod, h.pvc, h.node = r.fetchKubeObjects(ctx, inst)\n\n // Active database calls.\n h.dbState = r.queryDatabase(ctx, h.pod)\n h.effectiveConfig = r.readEffectiveConfig(ctx, h.pod)\n\n // Values from the collector/metric.\n h.diskUsage = r.collector.Usage(h.pvc)\n h.storageOp = r.collector.InFlightOp(h.pvc)\n\n return h, nil\n}\n\nA few things about this are deliberate, and only look obvious after you've been burned once or twice.\nGather once, at the top. Every sub-decision in the reconcile reads from this one snapshot. We don't re-query the database in the middle of the loop, or read the disk usage again three functions deep. If we did, different parts of the same reconcile could see different versions of reality. This sounds like a small detail, but it changes the whole design.\nFor example, the database might be the leader when we check at the top and a replica by the time another helper checks again. Then you get decisions that are individually reasonable, but wrong together. We have a rule in the codebase against stashing state back onto this handler mid-reconcile to pass between steps, because it reintroduces exactly the inconsistency we gathered the snapshot to avoid. Making the reconcileHandler immutable is one way to enforce that rule in the type system instead of relying on code review.\nPartial failures are tolerated. Reaching the database can fail while the Kubernetes reads succeed. That's not always an error that aborts the reconcile. It's a measured fact: \"Postgres is currently unreachable.\" That itself is something to record in status. A control loop that gives up entirely whenever one sensor is unavailable is a control loop that's down a lot. We degrade instead. Think of a car. If the rain sensor for the wipers is broken, the whole car doesn't stop. You can still drive, but you need to turn on a few things yourself.\nOnce the data snapshot exists, the reconcile is a sequence of small, idempotent steps, each comparing one slice of desired against observed and acting to close the gap:func (r *reconcileHandler) reconcile(ctx context.Context) (reconcile.Result, error) {\n var rb results.Builder\n rb.Merge(r.reconcileConfigMap(ctx)) // push desired config\n rb.Merge(r.reconcileDatabase(ctx)) // reload/restart if params drifted\n rb.Merge(r.reconcilePVC(ctx)) // grow the disk if needed\n rb.Merge(r.reconcilePod(ctx)) // create/replace the Pod\n rb.Merge(r.reconcileStatus(ctx)) // always last: write what we observed\n return rb.Result()\n}\n\nStatus is written last on purpose, because it's the measured variable: you record what's true after you've taken your actions and observed the result. Again, in our operators, it's not possible to write the status mid-reconcile.\nEach step is independently idempotent. Each returns a result, either \"I'm done\" or \"requeue me in 30 seconds, I'm waiting on something,\" and the results merge. It reads almost exactly like the body of my shell loop. The difference is that \"observe the state\" grew from df and pg_isready into a fan-in across multiple systems, and \"take an action\" grew from ssh into typed, conflict-aware API writes.\nThis is the operator. The kubelet, the scheduler, CSI, and CNI are infrastructure we get by using Kubernetes. This loop, with its messy real-world observe step, is the part we actually write and deal with. Because we know how the underlying system works, we can design it without treating Kubernetes like a black box.\nNot every edge comes from the API server\nThere's one more piece, and it lets me close a loop from Part 1 that I left deliberately: the disk-usage check.\nMy shell script polled df on every node every five seconds. For three nodes, fine. For thousands of databases, you can't reconcile every one of them every few seconds just to check a number that rarely changes; you'd spend all your CPU re-deriving \"still at 40%, still at 40%, still at 40%.\" This is the level-triggered model's one real cost: re-checking everything is correct, but it isn't free.\nThe fix is to add a sensor that emits its own edges. A background collector polls our metrics pipeline for disk usage on a slow cadence, keeps the last value per volume in memory, and only emits an event when usage crosses a threshold, not while it sits above or below one:// Edge detection. We fire only on the transition across the threshold,\n// not every cycle we happen to be above it. Hovering at 81% is silent;\n// crossing 80% upward is an event.\ncrossedUp := previousUsage < pvc.GrowThreshold && usage >= pvc.GrowThreshold\nif crossedUp {\n // -> generic event -> work queue -> reconcile\n relay.Send(Event{Key: pvc.Key})\n}\n\nThat event goes into the same work queue as the API watch events and triggers a normal reconcile of the affected instance. Same rule as before: the event wakes us up, the reconcile decides from the current state.\nFor example, say we have a 10GiB disk and it's using 8GiB. The collector saw it cross the threshold, so it wakes the reconciler. The reconciler reads the current usage, sees that it crossed the 80% threshold, and sets a new size on the PVC. After that, CSI handles the rest.\nAnd because edges can be missed (the collector could be down, an event could be dropped from a full channel), there's a resyncer: a periodic timer that enqueues every object for reconcile every minute or so, regardless of events. It's the safety net. It's our sleep 5 loop. There's also RequeueAfter, which a reconcile returns to say \"wake me again in 30 seconds,\" the controller's way of polling a slow external operation without holding a worker.\nThere are two more questions: how often should the loop run, and who is allowed to run it? Control theory calls the first one the sampling interval. The rule of thumb: act faster than the thing you're tracking changes, but not faster than it can respond. Reconciling a disk that fills over hours every few milliseconds just burns CPU to learn the same thing again.\nSo the operator puts boundaries around it.\nA coalescing delay handles noisy edge events: a burst of events for one object becomes one reconcile (think of it like a fan-in), not a thousand.\nThe resyncer is the safety net: every object gets looked at once in a while, even when nothing fires.\nAnd leader election answers the who: only one copy of the operator runs the loop at a time. Two controllers writing to the same database object is not \"more reliable.\" Even with idempotent controllers, they'll be requeueing due to conflicts and consuming unnecessary compute. In theory, a perfectly written controller should tolerate this. In practice, software is rarely perfect, and the safer boundary is worth it.\nTo close out Part 2, let me redraw the control loop again. The diagram in Part 1 had a few basic boxes. Now, the same loop represents a closed feedback loop more realistically:\n\nThere is one new arrow in this diagram: disturbances. A controller has two jobs. The first is setpoint tracking: someone edits the .spec, and the loop chases the new intent. The second is disturbance rejection: the world changes on its own. A node dies, a customer starts a bulk import, someone deletes a Pod by hand. The level-triggered reconcile treats both the same way: it only sees the gap.\nOur controller doesn't always touch Postgres directly. Sometimes it writes a PVC and lets CSI do the storage work. Sometimes it creates a Pod and lets the scheduler and kubelet do their part. This is what a production operator looks like: one loop we write, surrounded by other loops we don't write.\nNotice that every decision in this loop has been binary: start the Pod or don't, grow the disk or don't, rewrite the config or don't. That's an on/off controller, and it covers most of what an operator does. But not every question is yes/no; once the answer becomes how much rather than whether, you need a controller with memory and a sense of trend: how long you've been off, and how fast it's changing. That's a separate post.\nConclusion\nAll of this works, and most of the time it runs without anyone watching it. But the abstractions still leak, and they usually leak at a bad time.\nEventual consistency and the split between cache reads and API writes mean that a freshly-created object might not be visible to the thing that just created it. When something goes wrong, we're debugging Kubernetes objects, database state, metrics, volume operations, and sometimes the cloud provider at the same time. The bug is usually not in one clean place.\nThe declarative model is wonderful until it meets an operation that's inherently imperative and stateful, like a failover, a major-version upgrade, or a data migration. Then you have to turn a blocking, non-idempotent action into an idempotent one. That's a whole other blog post.\nThat complexity is easy to underestimate. If you're not dealing with sophisticated systems, if you can sacrifice availability, or if you don't care about scalability, maybe all this machinery isn't needed at all. Operators do not remove complexity. They move it into code someone has to understand.\nI still think it's worth it. For running thousands of databases that have to heal themselves without anyone watching, I don't know a better alternative. The hard parts are hard because the problem is hard, not because Kubernetes made it hard.\nKubernetes is not only a container runtime. It's not only a YAML processor, or an orchestrator, or whatever word we use that year. For me, the useful way to read Kubernetes is this: Kubernetes is a framework for feedback controllers, plus a consistent store to hold their setpoints and a shared event bus to wake them up.\nOnce you see that, the rest fits together. The kubelet, the scheduler, CSI, and your operator all read and write facts onto shared objects, and each one tries to move its own small part of the system toward the desired state. The core idea is still the same one we started with: write down what you want, look at what exists, make the next change, and repeat. Events wake the loop up, but the current state decides what happens.\nKubernetes didn't invent these ideas; a thermostat had them long before us. The mapping to control theory is not perfect, and some boundaries are fuzzy. But the core idea holds. We are writing feedback loops in Go and applying them to databases. Mechanical and electrical engineers figured out how to build stable, long-running systems before us. Software engineering is still catching up, and Kubernetes gives us a practical way to use those ideas in production.","aside","toc",[57,58,59],"title","The feedback loops behind Kubernetes","authors",[52],"categories",[51],"excerpt","Kubernetes is a framework for feedback controllers: write down what you want, observe what exists, make the next change, and repeat.","createdAt","2026-06-16","meta",{"_46":47,"_48":41,"_49":50,"_34":35},"canonical","https://planetscale.com/blog/the-feedback-loops-behind-kubernetes","description","image","/assets/the-feedback-loops-behind-kubernetes-social-DXEE2U-o.png","engineering",{"_25":53,"_54":55,"_56":53},"fatih","name","Fatih Arslan","x",{"_60":103,"_62":104,"_64":65,"_34":105},{"_60":67,"_62":68,"_64":65,"_34":69},{"_60":61,"_62":63,"_64":65,"_34":66},"children",[],"id","conclusion","level",2,"Conclusion",[70,71,72,73,74,75,76,77],"part-2-how-we-reinvented-kubernetes","Part 2: how we reinvented Kubernetes",{"_60":100,"_62":101,"_64":80,"_34":102},{"_60":97,"_62":98,"_64":80,"_34":99},{"_60":94,"_62":95,"_64":80,"_34":96},{"_60":91,"_62":92,"_64":80,"_34":93},{"_60":88,"_62":89,"_64":80,"_34":90},{"_60":85,"_62":86,"_64":80,"_34":87},{"_60":82,"_62":83,"_64":80,"_34":84},{"_60":78,"_62":79,"_64":80,"_34":81},[],"not-every-edge-comes-from-the-api-server",3,"Not every edge comes from the API server",[],"what-observe-actually-means-in-a-real-operator","What \"observe\" actually means in a real operator",[],"self-healing-by-design","Self-healing by design",[],"setpoint-and-measured-variable-spec-and-status","Setpoint and measured variable: spec and status",[],"informers-the-work-queue-and-a-cache","Informers, the work queue, and a cache",[],"edge-triggered-notifications-level-triggered-logic","Edge-triggered notifications, level-triggered logic",[],"the-for-loop-translated-to-kubernetes","The for-loop translated to Kubernetes",[],"the-other-loops","The other loops",[106,107,108,109,110,111,112,113],"part-1-running-postgres-by-hand","Part 1: running Postgres by hand",{"_60":135,"_62":136,"_64":80,"_34":137},{"_60":132,"_62":133,"_64":80,"_34":134},{"_60":129,"_62":130,"_64":80,"_34":131},{"_60":126,"_62":127,"_64":80,"_34":128},{"_60":123,"_62":124,"_64":80,"_34":125},{"_60":120,"_62":121,"_64":80,"_34":122},{"_60":117,"_62":118,"_64":80,"_34":119},{"_60":114,"_62":115,"_64":80,"_34":116},[],"what-we-actually-built","What we actually built",[],"changing-a-parameter","Changing a parameter",[],"the-watchdog-script","The watchdog script",[],"they-have-to-find-each-other","They have to find each other",[],"one-isnt-enough","One isn't enough",[],"it-needs-a-real-disk","It needs a real disk",[],"pick-a-node-by-hand","Pick a node, by hand",[],"one-container-one-machine","One container, one machine",["SingleFetchClassInstance",2208],["SingleFetchClassInstance",2204],["SingleFetchClassInstance",2200],["SingleFetchClassInstance",2196],["SingleFetchClassInstance",2159],["SingleFetchClassInstance",2156],["SingleFetchClassInstance",2148],["SingleFetchClassInstance",2140],["SingleFetchClassInstance",2136],["SingleFetchClassInstance",2132],["SingleFetchClassInstance",2128],["SingleFetchClassInstance",2114],["SingleFetchClassInstance",2106],["SingleFetchClassInstance",2091],["SingleFetchClassInstance",2087],["SingleFetchClassInstance",2078],["SingleFetchClassInstance",2070],["SingleFetchClassInstance",2066],["SingleFetchClassInstance",2062],["SingleFetchClassInstance",2058],["SingleFetchClassInstance",2050],["SingleFetchClassInstance",2024],["SingleFetchClassInstance",2015],["SingleFetchClassInstance",2007],["SingleFetchClassInstance",2003],["SingleFetchClassInstance",1983],["SingleFetchClassInstance",1978],["SingleFetchClassInstance",1974],["SingleFetchClassInstance",1952],["SingleFetchClassInstance",1944],["SingleFetchClassInstance",1940],["SingleFetchClassInstance",1917],["SingleFetchClassInstance",1913],["SingleFetchClassInstance",1909],["SingleFetchClassInstance",1899],["SingleFetchClassInstance",1890],["SingleFetchClassInstance",1882],["SingleFetchClassInstance",1856],["SingleFetchClassInstance",1852],["SingleFetchClassInstance",1847],["SingleFetchClassInstance",1824],["SingleFetchClassInstance",1816],["SingleFetchClassInstance",1807],["SingleFetchClassInstance",1792],["SingleFetchClassInstance",1782],["SingleFetchClassInstance",1759],["SingleFetchClassInstance",1749],["SingleFetchClassInstance",1658],["SingleFetchClassInstance",1643],["SingleFetchClassInstance",1639],["SingleFetchClassInstance",1606],["SingleFetchClassInstance",1602],["SingleFetchClassInstance",1599],["SingleFetchClassInstance",1591],["SingleFetchClassInstance",1587],["SingleFetchClassInstance",1579],["SingleFetchClassInstance",1575],["SingleFetchClassInstance",1561],["SingleFetchClassInstance",1540],["SingleFetchClassInstance",1487],["SingleFetchClassInstance",1459],["SingleFetchClassInstance",1450],["SingleFetchClassInstance",1427],["SingleFetchClassInstance",1417],["SingleFetchClassInstance",1413],["SingleFetchClassInstance",1405],["SingleFetchClassInstance",1389],["SingleFetchClassInstance",1385],["SingleFetchClassInstance",1381],["SingleFetchClassInstance",1373],["SingleFetchClassInstance",1369],["SingleFetchClassInstance",1336],["SingleFetchClassInstance",1332],["SingleFetchClassInstance",1328],["SingleFetchClassInstance",1324],["SingleFetchClassInstance",1311],["SingleFetchClassInstance",1307],["SingleFetchClassInstance",1292],["SingleFetchClassInstance",1288],["SingleFetchClassInstance",1279],["SingleFetchClassInstance",1239],["SingleFetchClassInstance",1216],["SingleFetchClassInstance",1206],["SingleFetchClassInstance",1198],["SingleFetchClassInstance",1194],["SingleFetchClassInstance",1178],["SingleFetchClassInstance",1168],["SingleFetchClassInstance",1132],["SingleFetchClassInstance",1110],["SingleFetchClassInstance",1101],["SingleFetchClassInstance",1097],["SingleFetchClassInstance",1093],["SingleFetchClassInstance",1089],["SingleFetchClassInstance",1073],["SingleFetchClassInstance",1069],["SingleFetchClassInstance",1065],["SingleFetchClassInstance",1061],["SingleFetchClassInstance",1038],["SingleFetchClassInstance",1021],["SingleFetchClassInstance",1017],["SingleFetchClassInstance",1013],["SingleFetchClassInstance",1009],["SingleFetchClassInstance",1001],["SingleFetchClassInstance",987],["SingleFetchClassInstance",969],["SingleFetchClassInstance",926],["SingleFetchClassInstance",910],["SingleFetchClassInstance",900],["SingleFetchClassInstance",878],["SingleFetchClassInstance",862],["SingleFetchClassInstance",854],["SingleFetchClassInstance",850],["SingleFetchClassInstance",835],["SingleFetchClassInstance",831],["SingleFetchClassInstance",792],["SingleFetchClassInstance",788],["SingleFetchClassInstance",757],["SingleFetchClassInstance",747],["SingleFetchClassInstance",739],["SingleFetchClassInstance",735],["SingleFetchClassInstance",731],["SingleFetchClassInstance",727],["SingleFetchClassInstance",684],["SingleFetchClassInstance",661],["SingleFetchClassInstance",657],["SingleFetchClassInstance",653],["SingleFetchClassInstance",649],["SingleFetchClassInstance",640],["SingleFetchClassInstance",630],["SingleFetchClassInstance",621],["SingleFetchClassInstance",617],["SingleFetchClassInstance",613],["SingleFetchClassInstance",609],["SingleFetchClassInstance",588],["SingleFetchClassInstance",584],["SingleFetchClassInstance",575],["SingleFetchClassInstance",571],["SingleFetchClassInstance",561],["SingleFetchClassInstance",557],["SingleFetchClassInstance",549],["SingleFetchClassInstance",545],["SingleFetchClassInstance",541],["SingleFetchClassInstance",520],["SingleFetchClassInstance",504],["SingleFetchClassInstance",500],["SingleFetchClassInstance",456],["SingleFetchClassInstance",452],["SingleFetchClassInstance",415],["SingleFetchClassInstance",384],["SingleFetchClassInstance",380],["SingleFetchClassInstance",357],["SingleFetchClassInstance",353],["SingleFetchClassInstance",342],["SingleFetchClassInstance",338],["SingleFetchClassInstance",334],["SingleFetchClassInstance",330],["SingleFetchClassInstance",326],["SingleFetchClassInstance",322],["SingleFetchClassInstance",311],["SingleFetchClassInstance",307],["SingleFetchClassInstance",299],{"_300":301,"_54":302,"_303":304,"_60":305},"$$mdtype","Tag","p","attributes",{},[306],"Kubernetes didn't invent these ideas; a thermostat had them long before us. The mapping to control theory is not perfect, and some boundaries are fuzzy. But the core idea holds. We are writing feedback loops in Go and applying them to databases. Mechanical and electrical engineers figured out how to build stable, long-running systems before us. Software engineering is still catching up, and Kubernetes gives us a practical way to use those ideas in production.",{"_300":301,"_54":302,"_303":308,"_60":309},{},[310],"Once you see that, the rest fits together. The kubelet, the scheduler, CSI, and your operator all read and write facts onto shared objects, and each one tries to move its own small part of the system toward the desired state. The core idea is still the same one we started with: write down what you want, look at what exists, make the next change, and repeat. Events wake the loop up, but the current state decides what happens.",{"_300":301,"_54":302,"_303":312,"_60":313},{},[314,315,316],"Kubernetes is not only a container runtime. It's not only a YAML processor, or an orchestrator, or whatever word we use that year. For me, the useful way to read Kubernetes is this: ",["SingleFetchClassInstance",317],", plus a consistent store to hold their setpoints and a shared event bus to wake them up.",{"_300":301,"_54":318,"_303":319,"_60":320},"strong",{},[321],"Kubernetes is a framework for feedback controllers",{"_300":301,"_54":302,"_303":323,"_60":324},{},[325],"I still think it's worth it. For running thousands of databases that have to heal themselves without anyone watching, I don't know a better alternative. The hard parts are hard because the problem is hard, not because Kubernetes made it hard.",{"_300":301,"_54":302,"_303":327,"_60":328},{},[329],"That complexity is easy to underestimate. If you're not dealing with sophisticated systems, if you can sacrifice availability, or if you don't care about scalability, maybe all this machinery isn't needed at all. Operators do not remove complexity. They move it into code someone has to understand.",{"_300":301,"_54":302,"_303":331,"_60":332},{},[333],"The declarative model is wonderful until it meets an operation that's inherently imperative and stateful, like a failover, a major-version upgrade, or a data migration. Then you have to turn a blocking, non-idempotent action into an idempotent one. That's a whole other blog post.",{"_300":301,"_54":302,"_303":335,"_60":336},{},[337],"Eventual consistency and the split between cache reads and API writes mean that a freshly-created object might not be visible to the thing that just created it. When something goes wrong, we're debugging Kubernetes objects, database state, metrics, volume operations, and sometimes the cloud provider at the same time. The bug is usually not in one clean place.",{"_300":301,"_54":302,"_303":339,"_60":340},{},[341],"All of this works, and most of the time it runs without anyone watching it. But the abstractions still leak, and they usually leak at a bad time.",{"_300":301,"_54":343,"_303":344,"_60":345},"h2",{"_62":63},[346],["SingleFetchClassInstance",347],{"_300":301,"_54":348,"_303":349,"_60":350},"a",{"_351":352},[66],"href","#conclusion",{"_300":301,"_54":354,"_303":355,"_60":356},"hr",{},[],{"_300":301,"_54":302,"_303":358,"_60":359},{},[360,361,362,363,364,365,366],"Notice that every decision in this loop has been binary: start the Pod or don't, grow the disk or don't, rewrite the config or don't. That's an ",["SingleFetchClassInstance",376],", and it covers most of what an operator does. But not every question is yes/no; once the answer becomes ",["SingleFetchClassInstance",372]," rather than ",["SingleFetchClassInstance",367],", you need a controller with memory and a sense of trend: how long you've been off, and how fast it's changing. That's a separate post.",{"_300":301,"_54":368,"_303":369,"_60":370},"em",{},[371],"whether",{"_300":301,"_54":368,"_303":373,"_60":374},{},[375],"how much",{"_300":301,"_54":368,"_303":377,"_60":378},{},[379],"on/off controller",{"_300":301,"_54":302,"_303":381,"_60":382},{},[383],"Our controller doesn't always touch Postgres directly. Sometimes it writes a PVC and lets CSI do the storage work. Sometimes it creates a Pod and lets the scheduler and kubelet do their part. This is what a production operator looks like: one loop we write, surrounded by other loops we don't write.",{"_300":301,"_54":302,"_303":385,"_60":386},{},[387,388,389,390,391,392,393,394,395],"There is one new arrow in this diagram: ",["SingleFetchClassInstance",411],". A controller has two jobs. The first is ",["SingleFetchClassInstance",406],": someone edits the ",["SingleFetchClassInstance",401],", and the loop chases the new intent. The second is ",["SingleFetchClassInstance",396],": the world changes on its own. A node dies, a customer starts a bulk import, someone deletes a Pod by hand. The level-triggered reconcile treats both the same way: it only sees the gap.",{"_300":301,"_54":348,"_303":397,"_60":398},{"_351":400},[399],"disturbance rejection","https://en.wikipedia.org/wiki/Control_theory",{"_300":301,"_54":402,"_303":403,"_60":404},"code",{},[405],".spec",{"_300":301,"_54":348,"_303":407,"_60":408},{"_351":410},[409],"setpoint tracking","https://en.wikipedia.org/wiki/Setpoint_%28control_system%29",{"_300":301,"_54":318,"_303":412,"_60":413},{},[414],"disturbances",{"_300":301,"_54":302,"_303":416,"_60":417},{},[418],["SingleFetchClassInstance",419],{"_300":301,"_54":420,"_303":421,"_60":422},"picture",{},[423,424,425],["SingleFetchClassInstance",448],["SingleFetchClassInstance",440],["SingleFetchClassInstance",426],{"_300":301,"_54":427,"_303":428,"_60":429},"img",{"_430":431,"_432":433,"_434":435,"_436":437,"_438":439},[],"alt","Production operator feedback loop diagram","height",943,"loading","lazy","src","https://planetscale-images.imgix.net/assets/part2-operator-loop-YZG6qYVG.png?auto=compress%2Cformat","width",1504,{"_300":301,"_54":441,"_303":442,"_60":443},"source",{"_444":445,"_446":447},[],"srcSet","https://planetscale-images.imgix.net/assets/part2-operator-loop-darkmode-jOHBdP3P.png?auto=compress%2Cformat","media","(prefers-color-scheme: dark)",{"_300":301,"_54":441,"_303":449,"_60":450},{"_444":437,"_446":451},[],"(prefers-color-scheme: light), (prefers-color-scheme: no-preference)",{"_300":301,"_54":302,"_303":453,"_60":454},{},[455],"To close out Part 2, let me redraw the control loop again. The diagram in Part 1 had a few basic boxes. Now, the same loop represents a closed feedback loop more realistically:",{"_300":301,"_54":457,"_303":458,"_60":459},"ul",{},[460,461,462],["SingleFetchClassInstance",490],["SingleFetchClassInstance",480],["SingleFetchClassInstance",463],{"_300":301,"_54":464,"_303":465,"_60":466},"li",{},[467,468,469,470,471],"And ",["SingleFetchClassInstance",476]," answers the ",["SingleFetchClassInstance",472],": only one copy of the operator runs the loop at a time. Two controllers writing to the same database object is not \"more reliable.\" Even with idempotent controllers, they'll be requeueing due to conflicts and consuming unnecessary compute. In theory, a perfectly written controller should tolerate this. In practice, software is rarely perfect, and the safer boundary is worth it.",{"_300":301,"_54":368,"_303":473,"_60":474},{},[475],"who",{"_300":301,"_54":318,"_303":477,"_60":478},{},[479],"leader election",{"_300":301,"_54":464,"_303":481,"_60":482},{},[483,484,485],"The ",["SingleFetchClassInstance",486]," is the safety net: every object gets looked at once in a while, even when nothing fires.",{"_300":301,"_54":318,"_303":487,"_60":488},{},[489],"resyncer",{"_300":301,"_54":464,"_303":491,"_60":492},{},[493,494,495],"A ",["SingleFetchClassInstance",496]," handles noisy edge events: a burst of events for one object becomes one reconcile (think of it like a fan-in), not a thousand.",{"_300":301,"_54":318,"_303":497,"_60":498},{},[499],"coalescing delay",{"_300":301,"_54":302,"_303":501,"_60":502},{},[503],"So the operator puts boundaries around it.",{"_300":301,"_54":302,"_303":505,"_60":506},{},[507,508,509,510,511],"There are two more questions: ",["SingleFetchClassInstance",516]," Control theory calls the first one the ",["SingleFetchClassInstance",512],". The rule of thumb: act faster than the thing you're tracking changes, but not faster than it can respond. Reconciling a disk that fills over hours every few milliseconds just burns CPU to learn the same thing again.",{"_300":301,"_54":368,"_303":513,"_60":514},{},[515],"sampling interval",{"_300":301,"_54":318,"_303":517,"_60":518},{},[519],"how often should the loop run, and who is allowed to run it?",{"_300":301,"_54":302,"_303":521,"_60":522},{},[523,524,525,526,527,528,529],"And because edges can be missed (the collector could be down, an event could be dropped from a full channel), there's a ",["SingleFetchClassInstance",538],": a periodic timer that enqueues every object for reconcile every minute or so, regardless of events. It's the safety net. It's our ",["SingleFetchClassInstance",534]," loop. There's also ",["SingleFetchClassInstance",530],", which a reconcile returns to say \"wake me again in 30 seconds,\" the controller's way of polling a slow external operation without holding a worker.",{"_300":301,"_54":402,"_303":531,"_60":532},{},[533],"RequeueAfter",{"_300":301,"_54":402,"_303":535,"_60":536},{},[537],"sleep 5",{"_300":301,"_54":318,"_303":539,"_60":540},{},[489],{"_300":301,"_54":302,"_303":542,"_60":543},{},[544],"For example, say we have a 10GiB disk and it's using 8GiB. The collector saw it cross the threshold, so it wakes the reconciler. The reconciler reads the current usage, sees that it crossed the 80% threshold, and sets a new size on the PVC. After that, CSI handles the rest.",{"_300":301,"_54":302,"_303":546,"_60":547},{},[548],"That event goes into the same work queue as the API watch events and triggers a normal reconcile of the affected instance. Same rule as before: the event wakes us up, the reconcile decides from the current state.",{"_300":301,"_54":550,"_303":551,"_60":552},"CodeBlock",{"_553":554,"_555":556},[],"html","

// Edge detection. We fire only on the transition across the threshold,\n// not every cycle we happen to be above it. Hovering at 81% is silent;\n// crossing 80% upward is an event.\ncrossedUp := previousUsage < pvc.GrowThreshold && usage >= pvc.GrowThreshold\nif crossedUp {\n    // -> generic event -> work queue -> reconcile\n    relay.Send(Event{Key: pvc.Key})\n}\n

","language","go",{"_300":301,"_54":302,"_303":558,"_60":559},{},[560],"The fix is to add a sensor that emits its own edges. A background collector polls our metrics pipeline for disk usage on a slow cadence, keeps the last value per volume in memory, and only emits an event when usage crosses a threshold, not while it sits above or below one:",{"_300":301,"_54":302,"_303":562,"_60":563},{},[564,565,566],"My shell script polled ",["SingleFetchClassInstance",567]," on every node every five seconds. For three nodes, fine. For thousands of databases, you can't reconcile every one of them every few seconds just to check a number that rarely changes; you'd spend all your CPU re-deriving \"still at 40%, still at 40%, still at 40%.\" This is the level-triggered model's one real cost: re-checking everything is correct, but it isn't free.",{"_300":301,"_54":402,"_303":568,"_60":569},{},[570],"df",{"_300":301,"_54":302,"_303":572,"_60":573},{},[574],"There's one more piece, and it lets me close a loop from Part 1 that I left deliberately: the disk-usage check.",{"_300":301,"_54":576,"_303":577,"_60":578},"h3",{"_62":79},[579],["SingleFetchClassInstance",580],{"_300":301,"_54":348,"_303":581,"_60":582},{"_351":583},[81],"#not-every-edge-comes-from-the-api-server",{"_300":301,"_54":302,"_303":585,"_60":586},{},[587],"This is the operator. The kubelet, the scheduler, CSI, and CNI are infrastructure we get by using Kubernetes. This loop, with its messy real-world observe step, is the part we actually write and deal with. Because we know how the underlying system works, we can design it without treating Kubernetes like a black box.",{"_300":301,"_54":302,"_303":589,"_60":590},{},[591,592,593,594,595,596,597],"Each step is independently idempotent. Each returns a result, either \"I'm done\" or \"requeue me in 30 seconds, I'm waiting on something,\" and the results merge. It reads almost exactly like the body of my shell loop. The difference is that \"observe the state\" grew from ",["SingleFetchClassInstance",606]," and ",["SingleFetchClassInstance",602]," into a fan-in across multiple systems, and \"take an action\" grew from ",["SingleFetchClassInstance",598]," into typed, conflict-aware API writes.",{"_300":301,"_54":402,"_303":599,"_60":600},{},[601],"ssh",{"_300":301,"_54":402,"_303":603,"_60":604},{},[605],"pg_isready",{"_300":301,"_54":402,"_303":607,"_60":608},{},[570],{"_300":301,"_54":302,"_303":610,"_60":611},{},[612],"Status is written last on purpose, because it's the measured variable: you record what's true after you've taken your actions and observed the result. Again, in our operators, it's not possible to write the status mid-reconcile.",{"_300":301,"_54":550,"_303":614,"_60":615},{"_553":616,"_555":556},[],"

func (r *reconcileHandler) reconcile(ctx context.Context) (reconcile.Result, error) {\n    var rb results.Builder\n    rb.Merge(r.reconcileConfigMap(ctx))   // push desired config\n    rb.Merge(r.reconcileDatabase(ctx))    // reload/restart if params drifted\n    rb.Merge(r.reconcilePVC(ctx))         // grow the disk if needed\n    rb.Merge(r.reconcilePod(ctx))         // create/replace the Pod\n    rb.Merge(r.reconcileStatus(ctx))      // always last: write what we observed\n    return rb.Result()\n}\n

",{"_300":301,"_54":302,"_303":618,"_60":619},{},[620],"Once the data snapshot exists, the reconcile is a sequence of small, idempotent steps, each comparing one slice of desired against observed and acting to close the gap:",{"_300":301,"_54":302,"_303":622,"_60":623},{},[624,625],["SingleFetchClassInstance",626]," Reaching the database can fail while the Kubernetes reads succeed. That's not always an error that aborts the reconcile. It's a measured fact: \"Postgres is currently unreachable.\" That itself is something to record in status. A control loop that gives up entirely whenever one sensor is unavailable is a control loop that's down a lot. We degrade instead. Think of a car. If the rain sensor for the wipers is broken, the whole car doesn't stop. You can still drive, but you need to turn on a few things yourself.",{"_300":301,"_54":318,"_303":627,"_60":628},{},[629],"Partial failures are tolerated.",{"_300":301,"_54":302,"_303":631,"_60":632},{},[633,634,635],"For example, the database might be the leader when we check at the top and a replica by the time another helper checks again. Then you get decisions that are individually reasonable, but wrong together. We have a rule in the codebase against stashing state back onto this handler mid-reconcile to pass between steps, because it reintroduces exactly the inconsistency we gathered the snapshot to avoid. Making the ",["SingleFetchClassInstance",636]," immutable is one way to enforce that rule in the type system instead of relying on code review.",{"_300":301,"_54":402,"_303":637,"_60":638},{},[639],"reconcileHandler",{"_300":301,"_54":302,"_303":641,"_60":642},{},[643,644],["SingleFetchClassInstance",645]," Every sub-decision in the reconcile reads from this one snapshot. We don't re-query the database in the middle of the loop, or read the disk usage again three functions deep. If we did, different parts of the same reconcile could see different versions of reality. This sounds like a small detail, but it changes the whole design.",{"_300":301,"_54":318,"_303":646,"_60":647},{},[648],"Gather once, at the top.",{"_300":301,"_54":302,"_303":650,"_60":651},{},[652],"A few things about this are deliberate, and only look obvious after you've been burned once or twice.",{"_300":301,"_54":550,"_303":654,"_60":655},{"_553":656,"_555":556},[],"

// The observation snapshot: everything we know about this instance, right now.\ntype reconcileHandler struct {\n    // The object (.spec = setpoint, .status = measured).\n    instance *v1.PostgresInstance\n\n    // Kubernetes objects.\n    pod  *corev1.Pod\n    pvc  *corev1.PersistentVolumeClaim\n    node *corev1.Node\n\n    // Database state.\n    dbState DatabaseState\n\n    // What Postgres actually loaded, not what we last wrote.\n    effectiveConfig map[string]string\n\n    // Collected out-of-band.\n    diskUsage *resource.Quantity\n\n    // The volume operation already in flight, if any.\n    storageOp *StorageOperation\n}\n\nfunc (r *Reconciler) newReconcileHandler(\n  ctx context.Context,\n  inst *v1.PostgresInstance,\n) (*reconcileHandler, error) {\n    h := &reconcileHandler{instance: inst}\n\n    // Cheap local reads.\n    h.pod, h.pvc, h.node = r.fetchKubeObjects(ctx, inst)\n\n    // Active database calls.\n    h.dbState         = r.queryDatabase(ctx, h.pod)\n    h.effectiveConfig = r.readEffectiveConfig(ctx, h.pod)\n\n    // Values from the collector/metric.\n    h.diskUsage = r.collector.Usage(h.pvc)\n    h.storageOp = r.collector.InFlightOp(h.pvc)\n\n    return h, nil\n}\n

",{"_300":301,"_54":302,"_303":658,"_60":659},{},[660],"In code, the snapshot is just a struct, and the reconcile's first move is to populate it. This is simplified, but faithful to the real shape:",{"_300":301,"_54":302,"_303":662,"_60":663},{},[664],["SingleFetchClassInstance",665],{"_300":301,"_54":420,"_303":666,"_60":667},{},[668,669,670],["SingleFetchClassInstance",681],["SingleFetchClassInstance",677],["SingleFetchClassInstance",671],{"_300":301,"_54":427,"_303":672,"_60":673},{"_430":674,"_432":675,"_434":435,"_436":676,"_438":439},[],"Observation fan-in for a reconciler",1007,"https://planetscale-images.imgix.net/assets/part2-observation-fan-in-DvyJkB0Y.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":678,"_60":679},{"_444":680,"_446":447},[],"https://planetscale-images.imgix.net/assets/part2-observation-fan-in-darkmode-KuH1rwY1.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":682,"_60":683},{"_444":676,"_446":451},[],{"_300":301,"_54":457,"_303":685,"_60":686},{},[687,688,689,690],["SingleFetchClassInstance",718],["SingleFetchClassInstance",709],["SingleFetchClassInstance",700],["SingleFetchClassInstance",691],{"_300":301,"_54":464,"_303":692,"_60":693},{},[694,695],["SingleFetchClassInstance",696]," Some signals are too expensive or too rate-limited to fetch on every reconcile: disk usage, or whether a volume operation we kicked off earlier is still in flight and where it sits in its cooldown window. A separate collector, often a background goroutine, gathers these on a slow cadence and keeps the last value per volume in memory. The reconcile reads that value instantly, without blocking on anything. Think of these as custom workqueues you implement.",{"_300":301,"_54":318,"_303":697,"_60":698},{},[699],"A background collector.",{"_300":301,"_54":464,"_303":701,"_60":702},{},[703,704],["SingleFetchClassInstance",705]," Its role, whether it's healthy, how far behind its followers are, whether it's currently accepting writes. Some of this comes from the agents that sit next to the database and manage it; some we get by opening a connection and asking the database directly. These calls carry a tight timeout and are allowed to fail, more on that below.",{"_300":301,"_54":318,"_303":706,"_60":707},{},[708],"The database's own view of its health.",{"_300":301,"_54":464,"_303":710,"_60":711},{},[712,713],["SingleFetchClassInstance",714]," Not what we last wrote down, but what the server has actually loaded, so we can compare the two and detect drift. Other entities can rewrite or reload the config on disk without us knowing, so the only honest source of truth is the running server itself, never our last write.",{"_300":301,"_54":318,"_303":715,"_60":716},{},[717],"The database's effective configuration.",{"_300":301,"_54":464,"_303":719,"_60":720},{},[721,722],["SingleFetchClassInstance",723],": the Pod, its PVC, the PV behind it, the Node it's on, the ConfigMap holding its config. These are the cheap local reads, the stuff we already talked about.",{"_300":301,"_54":318,"_303":724,"_60":725},{},[726],"The Kubernetes cache",{"_300":301,"_54":302,"_303":728,"_60":729},{},[730],"The sources gathered at the top of every reconcile:",{"_300":301,"_54":302,"_303":732,"_60":733},{},[734],"When an operator I work on reconciles a single Postgres instance, the first thing it does, before it decides anything, is build a snapshot of reality from every source that knows something true about that instance. Not just Kubernetes. Kubernetes barely knows anything about whether Postgres is actually healthy.",{"_300":301,"_54":302,"_303":736,"_60":737},{},[738],"Up to here I've been a little vague about the \"measure\" step, because in the examples the measured state is just \"list the child Pods.\" But in a real database operator it's a lot more than that.",{"_300":301,"_54":576,"_303":740,"_60":741},{"_62":83},[742],["SingleFetchClassInstance",743],{"_300":301,"_54":348,"_303":744,"_60":745},{"_351":746},[84],"#what-observe-actually-means-in-a-real-operator",{"_300":301,"_54":302,"_303":748,"_60":749},{},[750,751,752],"So we wrote ",["SingleFetchClassInstance",753]," and maybe thought we're good. Kubernetes turns that one line into several independent controllers that have never heard of each other, but still cooperate because they share the API server and watch each other's objects. I like this part a lot. Nobody calls a central orchestrator. Nobody passes a private message. The system heals itself.",{"_300":301,"_54":402,"_303":754,"_60":755},{},[756],"if ! pg_isready; then docker start; fi",{"_300":301,"_54":302,"_303":758,"_60":759},{},[760,761,762,763,764,765,766,767,768],"Control theory has a name for loops stacked like this: ",["SingleFetchClassInstance",779],". The output of an outer loop becomes the setpoint of an inner loop. A controller never writes its own ",["SingleFetchClassInstance",776],", but it writes ",["SingleFetchClassInstance",772]," objects' ",["SingleFetchClassInstance",769]," all the time. My operator writes the PVC's spec, and that spec is the setpoint the CSI controllers converge to. Each loop worries only about its own layer and trusts the loop below.",{"_300":301,"_54":402,"_303":770,"_60":771},{},[405],{"_300":301,"_54":368,"_303":773,"_60":774},{},[775],"other",{"_300":301,"_54":402,"_303":777,"_60":778},{},[405],{"_300":301,"_54":348,"_303":780,"_60":781},{"_351":787},[782],["SingleFetchClassInstance",783],{"_300":301,"_54":318,"_303":784,"_60":785},{},[786],"cascade control","https://en.wikipedia.org/wiki/Proportional%E2%80%93integral%E2%80%93derivative_controller#Cascade_control",{"_300":301,"_54":302,"_303":789,"_60":790},{},[791],"That's multiple feedback loops, each watching the layer below, each reacting to an edge and converging to its own level, chained together through the API server with nobody orchestrating the whole thing.",{"_300":301,"_54":793,"_303":794,"_60":795},"ol",{},[796,797,798,799,800,801,802],["SingleFetchClassInstance",827],["SingleFetchClassInstance",823],["SingleFetchClassInstance",819],["SingleFetchClassInstance",815],["SingleFetchClassInstance",811],["SingleFetchClassInstance",807],["SingleFetchClassInstance",803],{"_300":301,"_54":464,"_303":804,"_60":805},{},[806],"The kubelet gets triggered because that's an edge for that node's kubelet and it starts the container.",{"_300":301,"_54":464,"_303":808,"_60":809},{},[810],"The scheduler detects the unscheduled Pod, assigns a node.",{"_300":301,"_54":464,"_303":812,"_60":813},{},[814],"The replacement is an unscheduled Pod, an edge for the scheduler.",{"_300":301,"_54":464,"_303":816,"_60":817},{},[818],"The parent reconciles, observes its children (level-triggered), sees one is missing and the count is below the setpoint, and creates a replacement.",{"_300":301,"_54":464,"_303":820,"_60":821},{},[822],"Through the ownership link, that edge becomes a reconcile request for the parent.",{"_300":301,"_54":464,"_303":824,"_60":825},{},[826],"The Pod's deletion is a watch event, an edge.",{"_300":301,"_54":464,"_303":828,"_60":829},{},[830],"A node dies and takes a Pod with it.",{"_300":301,"_54":302,"_303":832,"_60":833},{},[834],"Here is an example. Follow the loop:",{"_300":301,"_54":302,"_303":836,"_60":837},{},[838,839,840,841,842],"When a controller creates a child object (a Pod, a PVC), it stamps an ",["SingleFetchClassInstance",847]," on the child pointing back at the parent. That reference does two things. It sets up garbage collection: delete the parent, and Kubernetes can cascade the delete to its children. And it gives the controller a way to map child changes back to the parent: \"when any object I own changes, enqueue my parent for a reconcile.\" ",["SingleFetchClassInstance",843]," allows you to link controllers to each other and create chains. If done right, all your controllers and systems fit together.",{"_300":301,"_54":402,"_303":844,"_60":845},{},[846],"ownerReference",{"_300":301,"_54":318,"_303":848,"_60":849},{},[846],{"_300":301,"_54":302,"_303":851,"_60":852},{},[853],"This is the part I like most.",{"_300":301,"_54":576,"_303":855,"_60":856},{"_62":86},[857],["SingleFetchClassInstance",858],{"_300":301,"_54":348,"_303":859,"_60":860},{"_351":861},[87],"#self-healing-by-design",{"_300":301,"_54":302,"_303":863,"_60":864},{},[865,866,867,868,869],"This is why the reconcile is ",["SingleFetchClassInstance",874],", and why that matters. Our shell script kept \"am I mid-failover?\" in a variable that died with the process. A Kubernetes controller keeps nothing important in memory. Every fact it needs is on an API object: the spec it's driving toward, the status it last observed, the conditions describing where things stand. Kill the controller, restart it on another node, and it picks up exactly where it left off, not because it saved its progress, but because there was never any in-memory progress to lose. The state lives in the cluster (API server, ",["SingleFetchClassInstance",870]," is what holds the state). The controller is just the loop that reads it.",{"_300":301,"_54":402,"_303":871,"_60":872},{},[873],"etcd",{"_300":301,"_54":318,"_303":875,"_60":876},{},[877],"stateless",{"_300":301,"_54":302,"_303":879,"_60":880},{},[881],["SingleFetchClassInstance",882],{"_300":301,"_54":420,"_303":883,"_60":884},{},[885,886,887],["SingleFetchClassInstance",897],["SingleFetchClassInstance",893],["SingleFetchClassInstance",888],{"_300":301,"_54":427,"_303":889,"_60":890},{"_430":891,"_432":675,"_434":435,"_436":892,"_438":439},[],"Spec and status as setpoint and measured variable","https://planetscale-images.imgix.net/assets/part2-spec-status-CinHPict.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":894,"_60":895},{"_444":896,"_446":447},[],"https://planetscale-images.imgix.net/assets/part2-spec-status-darkmode-DI9X1oFB.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":898,"_60":899},{"_444":892,"_446":451},[],{"_300":301,"_54":302,"_303":901,"_60":902},{},[903,904,905],"When ",["SingleFetchClassInstance",906],", the status you're looking at does not reflect the latest setpoint yet. That one comparison is how you tell \"converged\" from \"still working on it.\"",{"_300":301,"_54":402,"_303":907,"_60":908},{},[909],"observedGeneration < generation",{"_300":301,"_54":302,"_303":911,"_60":912},{},[913,914,915,916,917],"That split is the whole declarative model in two fields. It comes with a piece of bookkeeping that's pure control theory: ",["SingleFetchClassInstance",922]," increments when desired state changes, and by convention the controller writes back ",["SingleFetchClassInstance",918]," to say \"the state I'm reporting reflects this version of your intent.\"",{"_300":301,"_54":402,"_303":919,"_60":920},{},[921],".status.observedGeneration",{"_300":301,"_54":402,"_303":923,"_60":924},{},[925],".metadata.generation",{"_300":301,"_54":302,"_303":927,"_60":928},{},[929,930,931,932,933,934,935,936,937,938,939,940,941,942],["SingleFetchClassInstance",966]," is the ",["SingleFetchClassInstance",962],", the observed state. It's owned by the controller, written through a separate status subresource, and it's where you record what's actually true. The better the status, the better the controller can decide. A good ",["SingleFetchClassInstance",959]," field is what makes a controller pleasant to operate. The word ",["SingleFetchClassInstance",955]," comes from control theory; ",["SingleFetchClassInstance",950]," around 1960 to ask whether you can infer a system's internal state from its outputs. ",["SingleFetchClassInstance",947]," is also your response to any third-party system. If someone wants to learn the outcome of your actions, ",["SingleFetchClassInstance",943]," is the place to look at.",{"_300":301,"_54":402,"_303":944,"_60":945},{},[946],".status",{"_300":301,"_54":402,"_303":948,"_60":949},{},[946],{"_300":301,"_54":348,"_303":951,"_60":952},{"_351":954},[953],"Kalman coined it","https://en.wikipedia.org/wiki/Observability",{"_300":301,"_54":368,"_303":956,"_60":957},{},[958],"observability",{"_300":301,"_54":402,"_303":960,"_60":961},{},[946],{"_300":301,"_54":318,"_303":963,"_60":964},{},[965],"measured variable",{"_300":301,"_54":402,"_303":967,"_60":968},{},[946],{"_300":301,"_54":302,"_303":970,"_60":971},{},[972,930,973,974,975,976],["SingleFetchClassInstance",984],["SingleFetchClassInstance",980],", the desired state. It's owned by whoever created the object (a human, or another controller), and the reconciler treats it as read-only intent. It's an anti-pattern to write to the ",["SingleFetchClassInstance",977]," from inside the controller. If you do it, stop reading, go and fix your codebase. There are only a handful of exceptions, but a controller should generally never set its own setpoint.",{"_300":301,"_54":402,"_303":978,"_60":979},{},[405],{"_300":301,"_54":318,"_303":981,"_60":982},{},[983],"setpoint",{"_300":301,"_54":402,"_303":985,"_60":986},{},[405],{"_300":301,"_54":302,"_303":988,"_60":989},{},[990,991,593,992,993],"Back to the control diagram. My script kept its setpoint in shell variables and its measured state in the output of ",["SingleFetchClassInstance",998],["SingleFetchClassInstance",994],". Kubernetes gives both a permanent home, on the object itself.",{"_300":301,"_54":402,"_303":995,"_60":996},{},[997],"psql",{"_300":301,"_54":402,"_303":999,"_60":1000},{},[570],{"_300":301,"_54":576,"_303":1002,"_60":1003},{"_62":89},[1004],["SingleFetchClassInstance",1005],{"_300":301,"_54":348,"_303":1006,"_60":1007},{"_351":1008},[90],"#setpoint-and-measured-variable-spec-and-status",{"_300":301,"_54":302,"_303":1010,"_60":1011},{},[1012],"This is not only a Kubernetes issue. In any system with a read cache and a write-through path, read-after-write is not consistent unless you make it so. Most of the time the cache is exactly what you want: cheap, local, and eventually consistent. Eventual consistency is fine because the loop runs again. But the moment a decision would be destructive or non-idempotent if you acted on a stale read, you need to know which path you're on. Kubernetes solves many hard problems, but it also gives you a few new ones.",{"_300":301,"_54":550,"_303":1014,"_60":1015},{"_553":1016,"_555":556},[],"

// The cached client can be stale right after our own writes, which\n// would make us miscount and create duplicates. For this one read,\n// go direct to the API server instead of the cache. Slower,\n// but consistent for this decision.\nerr := r.apiReader.List(ctx, &instances, client.InNamespace(ns), labelSelector)\n

",{"_300":301,"_54":302,"_303":1018,"_60":1019},{},[1020],"The pragmatic one, which a lot of people use, is to bypass the cache for the reads where a stale view would cause a double-create or double-delete, and go straight to the API server:",{"_300":301,"_54":302,"_303":1022,"_60":1023},{},[1024,1025,1026,1027,1028],"There are two ways out. The correct one is the ",["SingleFetchClassInstance",1034],", the same trick the built-in ReplicaSet controller uses: you record that you expect to see N creations in memory, and you don't act again until the cache has caught up to your own writes. It works, but it's not easy to implement and it's a fair amount of machinery. Read more ",["SingleFetchClassInstance",1029],".",{"_300":301,"_54":348,"_303":1030,"_60":1031},{"_351":1033},[1032],"on Ahmet's blog","https://ahmet.im/blog/controller-pitfalls/",{"_300":301,"_54":318,"_303":1035,"_60":1036},{},[1037],"expectations pattern",{"_300":301,"_54":302,"_303":1039,"_60":1040},{},[1041],["SingleFetchClassInstance",1042],{"_300":301,"_54":420,"_303":1043,"_60":1044},{},[1045,1046,1047],["SingleFetchClassInstance",1058],["SingleFetchClassInstance",1054],["SingleFetchClassInstance",1048],{"_300":301,"_54":427,"_303":1049,"_60":1050},{"_430":1051,"_432":1052,"_434":435,"_436":1053,"_438":439},[],"Kubernetes cache reads versus API writes diagram",1071,"https://planetscale-images.imgix.net/assets/part2-cache-vs-api-4EYpZbTb.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1055,"_60":1056},{"_444":1057,"_446":447},[],"https://planetscale-images.imgix.net/assets/part2-cache-vs-api-darkmode-9mbMAIBr.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1059,"_60":1060},{"_444":1053,"_446":451},[],{"_300":301,"_54":302,"_303":1062,"_60":1063},{},[1064],"Now an event fires again a second later, before the cache has caught up with the Pods you just created. You list from the cache, and the new Pods aren't there yet. Your code decides it still needs to create them, and you create duplicates. This is the classic stale-cache double-create, and it's nasty because it only shows up under timing you can't reproduce on your laptop.",{"_300":301,"_54":550,"_303":1066,"_60":1067},{"_553":1068,"_555":556},[],"

// I want N replicas. I see fewer, so I create the missing ones.\nexisting, _ := r.listChildPods(ctx)        // reads the CACHE\nfor i := len(existing); i < desired; i++ {\n    r.client.Create(ctx, newPod(i))          // writes the API SERVER\n}\n

",{"_300":301,"_54":302,"_303":1070,"_60":1071},{},[1072],"Here is another edge case. Picture this sequence inside a reconcile:",{"_300":301,"_54":302,"_303":1074,"_60":1075},{},[1076,1077,1078,1079,1080],"Most of the time, what you want is to drop the call and ",["SingleFetchClassInstance",1085],". In the next reconcile, the ",["SingleFetchClassInstance",1081]," will see the updated object, and your write will never happen. That's how everything self-converges.",{"_300":301,"_54":402,"_303":1082,"_60":1083},{},[1084],"GET",{"_300":301,"_54":368,"_303":1086,"_60":1087},{},[1088],"requeue",{"_300":301,"_54":302,"_303":1090,"_60":1091},{},[1092],"If you write a field of an object and then read the same object again from the cache, the reconciler might think it's not updated yet. You write again, and you get a Conflict error. Retrying with a fresh read can be fine, but blindly retrying against the same stale cached view just spins.",{"_300":301,"_54":302,"_303":1094,"_60":1095},{},[1096],"Because of that, a read can be stale. You need to be prepared for this.",{"_300":301,"_54":302,"_303":1098,"_60":1099},{},[1100],"In controller-runtime, the client you're handed reads from the informer's local cache. Cache reads are cheap, they don't touch the API server, and that's how a controller reconciles thousands of objects without falling over. But your writes go straight to the API server. The cache only learns about your write when the resulting watch event comes back around, a moment later.",{"_300":301,"_54":302,"_303":1102,"_60":1103},{},[1104,1105],"Second, and this is a detail that bites people a lot: ",["SingleFetchClassInstance",1106],{"_300":301,"_54":318,"_303":1107,"_60":1108},{},[1109],"your reads and your writes in Kubernetes don't go to the same place.",{"_300":301,"_54":302,"_303":1111,"_60":1112},{},[1113],["SingleFetchClassInstance",1114],{"_300":301,"_54":420,"_303":1115,"_60":1116},{},[1117,1118,1119],["SingleFetchClassInstance",1129],["SingleFetchClassInstance",1125],["SingleFetchClassInstance",1120],{"_300":301,"_54":427,"_303":1121,"_60":1122},{"_430":1123,"_432":675,"_434":435,"_436":1124,"_438":439},[],"Informer, work queue, and controller diagram","https://planetscale-images.imgix.net/assets/part2-informer-queue-yyOXLWWC.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1126,"_60":1127},{"_444":1128,"_446":447},[],"https://planetscale-images.imgix.net/assets/part2-informer-queue-darkmode-ANBXf_4R.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1130,"_60":1131},{"_444":1124,"_446":451},[],{"_300":301,"_54":457,"_303":1133,"_60":1134},{},[1135,1136,1137],["SingleFetchClassInstance",1159],["SingleFetchClassInstance",1142],["SingleFetchClassInstance",1138],{"_300":301,"_54":464,"_303":1139,"_60":1140},{},[1141],"It lets you run a pool of workers pulling keys in parallel, which is your fan-out. Events fan in from the watch, collapse in the queue, and fan out to the workers. This is something you need to tune. The higher you set the pool, the more pressure you put on the system: more writes, more API calls, more load on the provider, and more CPU usage in the operator.",{"_300":301,"_54":464,"_303":1143,"_60":1144},{},[1145,1146,1147,1148,1149],"It ",["SingleFetchClassInstance",1155],": an object that keeps erroring backs off exponentially instead of spinning. This is ",["SingleFetchClassInstance",1150],", the same reason a crash-looping container backs off instead of restarting hot.",{"_300":301,"_54":348,"_303":1151,"_60":1152},{"_351":1154},[1153],"damping","https://en.wikipedia.org/wiki/Damping",{"_300":301,"_54":368,"_303":1156,"_60":1157},{},[1158],"rate-limits",{"_300":301,"_54":464,"_303":1160,"_60":1161},{},[1145,1162,1163],["SingleFetchClassInstance",1164],": if the same object is updated five times before you get to it, you reconcile it once, against the latest state (level-triggered again).",{"_300":301,"_54":368,"_303":1165,"_60":1166},{},[1167],"coalesces",{"_300":301,"_54":302,"_303":1169,"_60":1170},{},[1171,1172,1173],"First, the informer turns each watch event into a key and puts it on a ",["SingleFetchClassInstance",1174],". The queue does a lot of work for you.",{"_300":301,"_54":318,"_303":1175,"_60":1176},{},[1177],"work queue",{"_300":301,"_54":302,"_303":1179,"_60":1180},{},[1181,1182,1183,1184,1185],"The answer is the ",["SingleFetchClassInstance",1190],". An informer opens a single watch against the API server for a given resource type, streams every add, update, and delete, and keeps a complete in-memory ",["SingleFetchClassInstance",1186]," of the objects we're interested in. Two things matter here:",{"_300":301,"_54":318,"_303":1187,"_60":1188},{},[1189],"cache",{"_300":301,"_54":318,"_303":1191,"_60":1192},{},[1193],"informer",{"_300":301,"_54":302,"_303":1195,"_60":1196},{},[1197],"So where do the edges come from? And what stops a controller from DDoSing the API server by listing everything every five seconds like my script did?",{"_300":301,"_54":576,"_303":1199,"_60":1200},{"_62":92},[1201],["SingleFetchClassInstance",1202],{"_300":301,"_54":348,"_303":1203,"_60":1204},{"_351":1205},[93],"#informers-the-work-queue-and-a-cache",{"_300":301,"_54":302,"_303":1207,"_60":1208},{},[1209,1210,1211],"Our bash script stumbled into this property by accident, at least for the sake of the example. But the ",["SingleFetchClassInstance",1212]," framework gives it to you on purpose. It's why a Kubernetes controller can crash, get restarted ten minutes later, and converge correctly with no special recovery code. There is no recovery code. There is just the loop. The controller can reconstruct the world from scratch.",{"_300":301,"_54":402,"_303":1213,"_60":1214},{},[1215],"controller-runtime",{"_300":301,"_54":302,"_303":1217,"_60":1218},{},[1219],["SingleFetchClassInstance",1220],{"_300":301,"_54":420,"_303":1221,"_60":1222},{},[1223,1224,1225],["SingleFetchClassInstance",1236],["SingleFetchClassInstance",1232],["SingleFetchClassInstance",1226],{"_300":301,"_54":427,"_303":1227,"_60":1228},{"_430":1229,"_432":1230,"_434":435,"_436":1231,"_438":439},[],"Edge-triggered versus level-triggered scaling",927,"https://planetscale-images.imgix.net/assets/part2-edge-vs-level-BUiD8D-7.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1233,"_60":1234},{"_444":1235,"_446":447},[],"https://planetscale-images.imgix.net/assets/part2-edge-vs-level-darkmode-3DNb92aw.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1237,"_60":1238},{"_444":1231,"_446":451},[],{"_300":301,"_54":302,"_303":1240,"_60":1241},{},[1242,1243,1244,1245,1246,1247,1248,1249,1250,1251,1252,1253,1254],"Events (the edges) are only a hint that it's worth looking again. They tell you ",["SingleFetchClassInstance",1275]," to reconcile, never ",["SingleFetchClassInstance",1271]," to do. The reconcile itself is level-based: it reads the current state (e.g., ",["SingleFetchClassInstance",1267],") and drives toward the desired state (e.g., ",["SingleFetchClassInstance",1263],"), ignoring the triggering event completely. That's ",["SingleFetchClassInstance",1259]," ",["SingleFetchClassInstance",1255]," only gets a key. The framework makes it hard to write edge-triggered logic, on purpose. Edge-triggered logic is how you get a controller that's fragile and permanently wrong after its first hiccup.",{"_300":301,"_54":402,"_303":1256,"_60":1257},{},[1258],"Reconcile",{"_300":301,"_54":368,"_303":1260,"_60":1261},{},[1262],"why",{"_300":301,"_54":402,"_303":1264,"_60":1265},{},[1266],"create 2 replicas",{"_300":301,"_54":402,"_303":1268,"_60":1269},{},[1270],"replicas=3",{"_300":301,"_54":368,"_303":1272,"_60":1273},{},[1274],"what",{"_300":301,"_54":368,"_303":1276,"_60":1277},{},[1278],"when",{"_300":301,"_54":302,"_303":1280,"_60":1281},{},[1282,1283,1028],"Kubernetes controllers combine both: ",["SingleFetchClassInstance",1284],{"_300":301,"_54":318,"_303":1285,"_60":1286},{},[1287],"edge-triggered notifications, level-triggered logic",{"_300":301,"_54":302,"_303":1289,"_60":1290},{},[1291],"If you miss the event, no one cares. In the next reconcile loop you'll catch it. If your app crashes, it comes back, reads again and detects that it did not increase it yet, increases it.",{"_300":301,"_54":302,"_303":1293,"_60":1294},{},[1295,1296,1297,1298,1299],"The level-triggered model fixes all of that. Remember, our shell script never asked \"what changed?\" It asked \"what ",["SingleFetchClassInstance",1303]," true right now?\", every five seconds, from scratch. Miss a loop, and the next one catches up. Run the loop twice, and you get the same result. The current state of the world is the only input that matters, and it's always available to read. So in the level-triggered case, our example above becomes this: you read ",["SingleFetchClassInstance",1300],", you check the current number of replicas, which is 1, and you increase by 2.",{"_300":301,"_54":402,"_303":1301,"_60":1302},{},[1270],{"_300":301,"_54":368,"_303":1304,"_60":1305},{},[1306],"is",{"_300":301,"_54":302,"_303":1308,"_60":1309},{},[1310],"In the first case, you won't be able to self-correct. In the second case, if your handler blindly applies the delta again, you'll end up with 5 replicas (you overshoot), instead of 3.",{"_300":301,"_54":793,"_303":1312,"_60":1313},{},[1314,1315],["SingleFetchClassInstance",1320],["SingleFetchClassInstance",1316],{"_300":301,"_54":464,"_303":1317,"_60":1318},{},[1319],"You receive it twice.",{"_300":301,"_54":464,"_303":1321,"_60":1322},{},[1323],"You miss the event (maybe the queue dropped it, or the consumer, your app, dropped it due to a crash or a full buffer).",{"_300":301,"_54":302,"_303":1325,"_60":1326},{},[1327],"Here is a very concrete example. Assume you have 1 replica, and you increase it to 3 replicas. Because you have only subscribed to changes, either:",{"_300":301,"_54":302,"_303":1329,"_60":1330},{},[1331],"The problem is that this is very fragile. In distributed systems, if one component is fragile, the fragility spreads to the rest of the system. Why is edge-triggering fragile? Say your controller is down for thirty seconds. It misses the events from those thirty seconds, and its view of the world is now permanently wrong. If two events arrive out of order, you process them out of order. If an event is delivered twice, you act twice. You're rebuilding your state from a stream of events, and you've inherited all of event sourcing's hard problems.",{"_300":301,"_54":302,"_303":1333,"_60":1334},{},[1335],"My first mental model of controllers, and probably yours at some point, was edge-triggered: listen to a stream of changes, and for each change, try to converge.",{"_300":301,"_54":457,"_303":1337,"_60":1338},{},[1339,1340],["SingleFetchClassInstance",1360],["SingleFetchClassInstance",1341],{"_300":301,"_54":464,"_303":1342,"_60":1343},{},[1344,1345,1346,1347,1348,1349],["SingleFetchClassInstance",1356],": act on the current state, regardless of how you got there. \"The disk ",["SingleFetchClassInstance",1353]," at 85%.\" \"The Pod ",["SingleFetchClassInstance",1350]," missing.\" \"The number of replicas is 3.\"",{"_300":301,"_54":368,"_303":1351,"_60":1352},{},[1306],{"_300":301,"_54":368,"_303":1354,"_60":1355},{},[1306],{"_300":301,"_54":318,"_303":1357,"_60":1358},{},[1359],"Level-triggered",{"_300":301,"_54":464,"_303":1361,"_60":1362},{},[1363,1364],["SingleFetchClassInstance",1365],": act on transitions, on events. \"The disk crossed 80%.\" \"The Pod was deleted.\" \"The number of replicas increased by 2.\"",{"_300":301,"_54":318,"_303":1366,"_60":1367},{},[1368],"Edge-triggered",{"_300":301,"_54":302,"_303":1370,"_60":1371},{},[1372],"There are two ways to build any closed feedback loop:",{"_300":301,"_54":576,"_303":1374,"_60":1375},{"_62":95},[1376],["SingleFetchClassInstance",1377],{"_300":301,"_54":348,"_303":1378,"_60":1379},{"_351":1380},[96],"#edge-triggered-notifications-level-triggered-logic",{"_300":301,"_54":302,"_303":1382,"_60":1383},{},[1384],"Notice what is missing here: the function isn't told what changed. There is no diff. It isn't handed the old object and the new object. It isn't given an event type. It gets a key, a namespace and a name, and nothing else. It's minimal by design, because it has to work for many different controllers. The function's job is to fetch the object with that namespace/name, look at the world, and converge to the desired state.",{"_300":301,"_54":550,"_303":1386,"_60":1387},{"_553":1388,"_555":556},[],"

func (r *Reconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {\n    // req contains a namespace/name. That's it. That's the whole input.\n}\n

",{"_300":301,"_54":302,"_303":1390,"_60":1391},{},[1392,1393,1394,1395,1396],"In Kubernetes, our watchdog script is a ",["SingleFetchClassInstance",1401],", and the standard way to write one in Go is a library called ",["SingleFetchClassInstance",1397],". At its heart, it's a function with a basic signature:",{"_300":301,"_54":348,"_303":1398,"_60":1399},{"_351":1400},[1215],"https://github.com/kubernetes-sigs/controller-runtime",{"_300":301,"_54":318,"_303":1402,"_60":1403},{},[1404],"controller",{"_300":301,"_54":576,"_303":1406,"_60":1407},{"_62":98},[1408],["SingleFetchClassInstance",1409],{"_300":301,"_54":348,"_303":1410,"_60":1411},{"_351":1412},[99],"#the-for-loop-translated-to-kubernetes",{"_300":301,"_54":302,"_303":1414,"_60":1415},{},[1416],"That's the operator.",{"_300":301,"_54":302,"_303":1418,"_60":1419},{},[1420,1421,1422],"All of this so far is useful context, but the part we care about is ",["SingleFetchClassInstance",1423]," loop, because that's the one we get to write ourselves.",{"_300":301,"_54":368,"_303":1424,"_60":1425},{},[1426],"our watchdog",{"_300":301,"_54":302,"_303":1428,"_60":1429},{},[1430],["SingleFetchClassInstance",1431],{"_300":301,"_54":420,"_303":1432,"_60":1433},{},[1434,1435,1436],["SingleFetchClassInstance",1447],["SingleFetchClassInstance",1443],["SingleFetchClassInstance",1437],{"_300":301,"_54":427,"_303":1438,"_60":1439},{"_430":1440,"_432":1441,"_434":435,"_436":1442,"_438":439},[],"Mapping manual operations to Kubernetes controllers",883,"https://planetscale-images.imgix.net/assets/part2-kubernetes-mapping-BqGHE4E0.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1444,"_60":1445},{"_444":1446,"_446":447},[],"https://planetscale-images.imgix.net/assets/part2-kubernetes-mapping-darkmode-CAz9s2jY.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1448,"_60":1449},{"_444":1442,"_446":451},[],{"_300":301,"_54":302,"_303":1451,"_60":1452},{},[1453,1454,1455],"As you see, all the problems we solved with ",["SingleFetchClassInstance",1456]," and various scripts are replaced by Kubernetes components and drivers. And these are just a few of them:",{"_300":301,"_54":402,"_303":1457,"_60":1458},{},[601],{"_300":301,"_54":302,"_303":1460,"_60":1461},{},[1462,1463,1464,1465,1466,1467,1468,1469],["SingleFetchClassInstance",1483]," The ",["SingleFetchClassInstance",1479]," problem is solved at a layer we no longer have to think about. A CNI plugin gives Pods their network identity; Cilium, for example, does this with eBPF instead of a pile of ",["SingleFetchClassInstance",1475]," rules. For stateful workloads, a StatefulSet plus a ",["SingleFetchClassInstance",1470]," gives each replica its own stable DNS name, which is exactly what a Postgres replica needs. The hard-coded IP that broke our cluster becomes a name that keeps working. DNS is only one tool here; other service discovery systems like etcd, ZooKeeper, and Consul solve similar problems.",{"_300":301,"_54":348,"_303":1471,"_60":1472},{"_351":1474},[1473],"headless Service","https://kubernetes.io/docs/concepts/services-networking/service/#headless-services",{"_300":301,"_54":402,"_303":1476,"_60":1477},{},[1478],"iptables",{"_300":301,"_54":402,"_303":1480,"_60":1481},{},[1482],"/etc/hosts",{"_300":301,"_54":318,"_303":1484,"_60":1485},{},[1486],"Making them find each other: the CNI and Services.",{"_300":301,"_54":302,"_303":1488,"_60":1489},{},[1490,1491,1492,593,1493,1494,1495,1496,1497,1498,1499,1500,1501,1502,1503,1504],["SingleFetchClassInstance",1536]," My multi-step ",["SingleFetchClassInstance",1532],["SingleFetchClassInstance",1528]," script becomes a ",["SingleFetchClassInstance",1524],", which is a declarative request for storage. The ",["SingleFetchClassInstance",1519]," driver turns that request into a real volume. CSI itself is a set of controllers and sidecars: one provisions, one attaches, one resizes, and so on, while the kubelet calls the driver's node plugin to do the actual mount. It's a family of controllers. If there is a PVC but no disk behind it, one controller creates the disk. If the PVC size increases, another controller calls the provider API (e.g., AWS ",["SingleFetchClassInstance",1515],"). Again, I write intent, and a controller does the actual work. (note: I wrote one of the early production CSI drivers, ",["SingleFetchClassInstance",1510],", and a ",["SingleFetchClassInstance",1505],".)",{"_300":301,"_54":348,"_303":1506,"_60":1507},{"_351":1509},[1508],"long post about building one","https://arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/",{"_300":301,"_54":348,"_303":1511,"_60":1512},{"_351":1514},[1513],"csi-digitalocean","https://github.com/digitalocean/csi-digitalocean",{"_300":301,"_54":402,"_303":1516,"_60":1517},{},[1518],"ModifyVolume",{"_300":301,"_54":348,"_303":1520,"_60":1521},{"_351":1523},[1522],"Container Storage Interface (CSI)","https://github.com/container-storage-interface/spec/blob/master/spec.md",{"_300":301,"_54":402,"_303":1525,"_60":1526},{},[1527],"PersistentVolumeClaim",{"_300":301,"_54":402,"_303":1529,"_60":1530},{},[1531],"mount",{"_300":301,"_54":402,"_303":1533,"_60":1534},{},[1535],"mkfs",{"_300":301,"_54":318,"_303":1537,"_60":1538},{},[1539],"Attaching the disk: CSI and the PV/PVC sync.",{"_300":301,"_54":302,"_303":1541,"_60":1542},{},[1543,1544,1545,1546,1547,1548],["SingleFetchClassInstance",1557]," Remember me choosing ",["SingleFetchClassInstance",1553],"? That's the scheduler's whole reason to exist. It watches for Pods with no node assigned, filters out the nodes that can't work, scores the rest, and writes the decision to one field: ",["SingleFetchClassInstance",1549],". The scheduler doesn't start the container; it records the placement and lets the kubelet pick it up. You will realize that most things in Kubernetes are decoupled like this.",{"_300":301,"_54":402,"_303":1550,"_60":1551},{},[1552],"pod.Spec.NodeName",{"_300":301,"_54":402,"_303":1554,"_60":1555},{},[1556],"node-07",{"_300":301,"_54":318,"_303":1558,"_60":1559},{},[1560],"Picking a node: the scheduler.",{"_300":301,"_54":302,"_303":1562,"_60":1563},{},[1564,1565,1566,1567],["SingleFetchClassInstance",1571]," First, a quick definition: a Pod is the smallest thing Kubernetes runs, one or more containers scheduled together on a node and sharing its network. For us it's the Postgres container. On every node runs an agent called the kubelet. Its desired state is the set of Pods assigned to its node, which it learns from the API server. Its observed state is the set of containers actually running, which it gets from the container runtime. When they differ, it starts the missing container, kills the extra one, or restarts the crashed one. My ",["SingleFetchClassInstance",1568]," is the kubelet's job, just done properly. The kubelet doesn't shell into anything; it talks to containerd over a gRPC socket, which talks to runc.",{"_300":301,"_54":402,"_303":1569,"_60":1570},{},[756],{"_300":301,"_54":318,"_303":1572,"_60":1573},{},[1574],"Spinning up the container: the kubelet.",{"_300":301,"_54":302,"_303":1576,"_60":1577},{},[1578],"Let's go through some of the pieces we built by hand before the watchdog loop. You already know these components by name. What you might not have noticed is that they also work like controllers.",{"_300":301,"_54":576,"_303":1580,"_60":1581},{"_62":101},[1582],["SingleFetchClassInstance",1583],{"_300":301,"_54":348,"_303":1584,"_60":1585},{"_351":1586},[102],"#the-other-loops",{"_300":301,"_54":302,"_303":1588,"_60":1589},{},[1590],"Now we can map what we hand-rolled in Part 1 to Kubernetes. Almost all of it already exists there. The operator is the part we care about.",{"_300":301,"_54":343,"_303":1592,"_60":1593},{"_62":68},[1594],["SingleFetchClassInstance",1595],{"_300":301,"_54":348,"_303":1596,"_60":1597},{"_351":1598},[69],"#part-2-how-we-reinvented-kubernetes",{"_300":301,"_54":354,"_303":1600,"_60":1601},{},[],{"_300":301,"_54":302,"_303":1603,"_60":1604},{},[1605],"What if the script also fails? Who runs it then? We could keep hardening this script, but look at where it goes: we would need a real store for the desired state, watches instead of polling, a work queue, retries, leader election. We would be rebuilding Kubernetes. The real platform already exists, and it's Kubernetes.",{"_300":301,"_54":457,"_303":1607,"_60":1608},{},[1609,1610,1611,1612,1613],["SingleFetchClassInstance",1635],["SingleFetchClassInstance",1631],["SingleFetchClassInstance",1627],["SingleFetchClassInstance",1618],["SingleFetchClassInstance",1614],{"_300":301,"_54":464,"_303":1615,"_60":1616},{},[1617],"And the moment I want a second kind of resource, a connection pooler, a backup job, a read replica in another region, I'm copy-pasting this whole structure.",{"_300":301,"_54":464,"_303":1619,"_60":1620},{},[1621,1622,1623],"It has no idea what to do when the ",["SingleFetchClassInstance",1624]," itself times out.",{"_300":301,"_54":402,"_303":1625,"_60":1626},{},[601],{"_300":301,"_54":464,"_303":1628,"_60":1629},{},[1630],"It polls every node every five seconds whether anything changed or not, which is fine for three nodes, but too expensive for three thousand nodes.",{"_300":301,"_54":464,"_303":1632,"_60":1633},{},[1634],"It keeps its only real state, \"am I mid-failover?\", in a shell variable that could die with the process.",{"_300":301,"_54":464,"_303":1636,"_60":1637},{},[1638],"It has no concurrency control, so two copies of the script can race each other. Imagine both deciding to promote a different replica.",{"_300":301,"_54":302,"_303":1640,"_60":1641},{},[1642],"A Bash loop is not a production control plane. Just to name a few issues with it:",{"_300":301,"_54":302,"_303":1644,"_60":1645},{},[1646,1647,1648,1649,1650],"That also gives us a nice way to understand ",["SingleFetchClassInstance",1654]," control. My very first attempt, ",["SingleFetchClassInstance",1651]," in, run the command, and walk away, was open-loop: fire an action and assume it worked. The Bash script is closed-loop because it keeps feeding the measured state back into the next decision.",{"_300":301,"_54":402,"_303":1652,"_60":1653},{},[601],{"_300":301,"_54":318,"_303":1655,"_60":1656},{},[1657],"open-loop",{"_300":301,"_54":457,"_303":1659,"_60":1660},{},[1661,1662,1663,1664,1665,1666],["SingleFetchClassInstance",1741],["SingleFetchClassInstance",1718],["SingleFetchClassInstance",1709],["SingleFetchClassInstance",1695],["SingleFetchClassInstance",1676],["SingleFetchClassInstance",1667],{"_300":301,"_54":464,"_303":1668,"_60":1669},{},[483,1670,1671],["SingleFetchClassInstance",1672]," is the system being controlled, Postgres and its disk.",{"_300":301,"_54":318,"_303":1673,"_60":1674},{},[1675],"plant",{"_300":301,"_54":464,"_303":1677,"_60":1678},{},[483,1679,1680,1681,1682,1683,1028],["SingleFetchClassInstance",1691]," is what carries out the action: ",["SingleFetchClassInstance",1688]," plus ",["SingleFetchClassInstance",1684],{"_300":301,"_54":402,"_303":1685,"_60":1686},{},[1687],"docker start",{"_300":301,"_54":402,"_303":1689,"_60":1690},{},[601],{"_300":301,"_54":318,"_303":1692,"_60":1693},{},[1694],"actuator",{"_300":301,"_54":464,"_303":1696,"_60":1697},{},[483,1698,1699,1700,1701],["SingleFetchClassInstance",1706]," is the body of the loop, the ",["SingleFetchClassInstance",1702]," statements that decide what to do. It is not the whole script.",{"_300":301,"_54":402,"_303":1703,"_60":1704},{},[1705],"if",{"_300":301,"_54":318,"_303":1707,"_60":1708},{},[1404],{"_300":301,"_54":464,"_303":1710,"_60":1711},{},[483,1712,1713],["SingleFetchClassInstance",1714]," (e) is the difference between them.",{"_300":301,"_54":318,"_303":1715,"_60":1716},{},[1717],"error",{"_300":301,"_54":464,"_303":1719,"_60":1720},{},[483,1721,1722,1723,1724,1725,1724,1726,1028],["SingleFetchClassInstance",1737]," is what I observe: ",["SingleFetchClassInstance",1734],", ",["SingleFetchClassInstance",1731],["SingleFetchClassInstance",1727],{"_300":301,"_54":402,"_303":1728,"_60":1729},{},[1730],"show max_connections",{"_300":301,"_54":402,"_303":1732,"_60":1733},{},[570],{"_300":301,"_54":402,"_303":1735,"_60":1736},{},[605],{"_300":301,"_54":318,"_303":1738,"_60":1739},{},[1740],"measured output",{"_300":301,"_54":464,"_303":1742,"_60":1743},{},[483,1744,1745],["SingleFetchClassInstance",1746]," is my desired state, the variables at the top of the script (disk size, max_connections and so on).",{"_300":301,"_54":318,"_303":1747,"_60":1748},{},[983],{"_300":301,"_54":302,"_303":1750,"_60":1751},{},[1752,1753,1754],"Here is how the vocabulary from ",["SingleFetchClassInstance",1755]," maps cleanly onto my shell script:",{"_300":301,"_54":348,"_303":1756,"_60":1757},{"_351":400},[1758],"control theory",{"_300":301,"_54":302,"_303":1760,"_60":1761},{},[1762],["SingleFetchClassInstance",1763],{"_300":301,"_54":420,"_303":1764,"_60":1765},{},[1766,1767,1768],["SingleFetchClassInstance",1779],["SingleFetchClassInstance",1775],["SingleFetchClassInstance",1769],{"_300":301,"_54":427,"_303":1770,"_60":1771},{"_430":1772,"_432":1773,"_434":435,"_436":1774,"_438":439},[],"Closed feedback loop diagram",564,"https://planetscale-images.imgix.net/assets/part1-feedback-loop-Z0q4qAPX.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1776,"_60":1777},{"_444":1778,"_446":447},[],"https://planetscale-images.imgix.net/assets/part1-feedback-loop-darkmode-BeMyZLwB.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1780,"_60":1781},{"_444":1774,"_446":451},[],{"_300":301,"_54":302,"_303":1783,"_60":1784},{},[1785,1786,1787],"The nice part is that the same loop works for different problems. It can restart a dead process, grow a disk, or push ",["SingleFetchClassInstance",1788],". The action changes, but the shape stays the same: read what I want, observe what I have, compare them, act, repeat. If I draw the same thing as a block diagram, with the control theory names added, it would look like this:",{"_300":301,"_54":402,"_303":1789,"_60":1790},{},[1791],"max_connections = 500",{"_300":301,"_54":302,"_303":1793,"_60":1794},{},[1795,1796,1797,1798,1799],"That's a ",["SingleFetchClassInstance",1803],". The word \"closed\" matters. It means the output of the system is fed back into the next decision. I don't run ",["SingleFetchClassInstance",1800]," and assume the database is fine. I check the database again. If it is still wrong, I act again. If it is already correct, I do nothing.",{"_300":301,"_54":402,"_303":1801,"_60":1802},{},[1687],{"_300":301,"_54":318,"_303":1804,"_60":1805},{},[1806],"closed feedback loop",{"_300":301,"_54":302,"_303":1808,"_60":1809},{},[1810,1811,1812],"I started with a desired state that was written down in one place: three instances, this disk size, ",["SingleFetchClassInstance",1813],". Every few seconds I observe the actual state of the system. I compute the difference. I take whatever action closes that difference. Then I do it again, forever.",{"_300":301,"_54":402,"_303":1814,"_60":1815},{},[1791],{"_300":301,"_54":576,"_303":1817,"_60":1818},{"_62":115},[1819],["SingleFetchClassInstance",1820],{"_300":301,"_54":348,"_303":1821,"_60":1822},{"_351":1823},[116],"#what-we-actually-built",{"_300":301,"_54":302,"_303":1825,"_60":1826},{},[1827,1828,1829,1830,1831,1832,1833],"This is the same idea as before. I read what I ",["SingleFetchClassInstance",1843]," (a variable). Observe what I ",["SingleFetchClassInstance",1839]," (a query). If they differ, I take an action to close the difference. Again, I don't track whether I changed it last time. All I do is compare and ",["SingleFetchClassInstance",1834],", every loop.",{"_300":301,"_54":348,"_303":1835,"_60":1836},{"_351":1838},[1837],"converge","https://dictionary.cambridge.org/dictionary/english/converge",{"_300":301,"_54":368,"_303":1840,"_60":1841},{},[1842],"have",{"_300":301,"_54":368,"_303":1844,"_60":1845},{},[1846],"want",{"_300":301,"_54":550,"_303":1848,"_60":1849},{"_553":1850,"_555":1851},[],"

WANT_MAX_CONNECTIONS=500\n\nfor node in node-07 node-12 node-19; do\n  have=$(ssh \"$node\" \"psql -tAc 'show max_connections'\")\n  if [ \"$have\" != \"$WANT_MAX_CONNECTIONS\" ]; then\n    ssh \"$node\" \"sed -i 's/^max_connections.*/max_connections = $WANT_MAX_CONNECTIONS/' /var/lib/pg-data/postgresql.conf\"\n    ssh \"$node\" \"docker restart pg\"\n  fi\ndone\n

","bash",{"_300":301,"_54":302,"_303":1853,"_60":1854},{},[1855],"Because I know that ssh'ing into the nodes manually isn't a thing I want anymore, I do the same thing we did previously: I write the desired value down in one place and teach the loop to enforce it.",{"_300":301,"_54":302,"_303":1857,"_60":1858},{},[1859,1860,1861,1862,1863,1864,1865],"Let's make things a little more complex. I need to raise ",["SingleFetchClassInstance",1873]," from 100 to 500. This one is not a reload-only change. PostgreSQL says it can only be set at server start, so the manual version is to ",["SingleFetchClassInstance",1870]," into each box, edit ",["SingleFetchClassInstance",1866],", restart Postgres, and check that it took on all three.",{"_300":301,"_54":402,"_303":1867,"_60":1868},{},[1869],"postgresql.conf",{"_300":301,"_54":402,"_303":1871,"_60":1872},{},[601],{"_300":301,"_54":348,"_303":1874,"_60":1875},{"_351":1881},[1876],["SingleFetchClassInstance",1877],{"_300":301,"_54":402,"_303":1878,"_60":1879},{},[1880],"max_connections","https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-MAX-CONNECTIONS",{"_300":301,"_54":576,"_303":1883,"_60":1884},{"_62":118},[1885],["SingleFetchClassInstance",1886],{"_300":301,"_54":348,"_303":1887,"_60":1888},{"_351":1889},[119],"#changing-a-parameter",{"_300":301,"_54":302,"_303":1891,"_60":1892},{},[1893,1894,1028],"If a process is down, start it. If a disk is filling, grow it. Run the loop once or run it a thousand times and the result is the same, because each action is conditional on the current state. The script is ",["SingleFetchClassInstance",1895],{"_300":301,"_54":368,"_303":1896,"_60":1897},{},[1898],"idempotent",{"_300":301,"_54":302,"_303":1900,"_60":1901},{},[1902,1903,1904],"It's written in Bash, and probably has tons of bugs. You notice something here? The loop doesn't care ",["SingleFetchClassInstance",1905]," the database got into a bad state. Every five seconds it looks at the current state of the world and asks this question: does reality match what I want?",{"_300":301,"_54":368,"_303":1906,"_60":1907},{},[1908],"how",{"_300":301,"_54":550,"_303":1910,"_60":1911},{"_553":1912,"_555":1851},[],"

while true; do\n  for node in node-07 node-12 node-19; do\n    if ! ssh \"$node\" 'pg_isready -q'; then\n      ssh \"$node\" 'docker start pg'       # it died, bring it back\n    fi\n\n    usage=$(ssh \"$node\" \"df --output=pcent /var/lib/pg-data | tail -1 | tr -dc 0-9\")\n    if [ \"$usage\" -gt 80 ]; then\n      grow_volume \"$node\"                  # disk filling, make it bigger\n    fi\n  done\n  sleep 5\ndone\n

",{"_300":301,"_54":302,"_303":1914,"_60":1915},{},[1916],"Let's assume we've set up a simple uptime monitor and we're going to get paged for all these cases. To avoid getting paged at night, we do the sensible thing: write a script. So we decide to write a loop that wakes up every few seconds, looks at each node, and fixes whatever's wrong.",{"_300":301,"_54":457,"_303":1918,"_60":1919},{},[1920,1921,1922,1923],["SingleFetchClassInstance",1936],["SingleFetchClassInstance",1932],["SingleFetchClassInstance",1928],["SingleFetchClassInstance",1924],{"_300":301,"_54":464,"_303":1925,"_60":1926},{},[1927],"A config I changed on two nodes but forgot on the third one. They are now out of sync.",{"_300":301,"_54":464,"_303":1929,"_60":1930},{},[1931],"The primary fails and a replica has to be promoted.",{"_300":301,"_54":464,"_303":1933,"_60":1934},{},[1935],"A disk gets full.",{"_300":301,"_54":464,"_303":1937,"_60":1938},{},[1939],"A replica process dies and doesn't come back.",{"_300":301,"_54":302,"_303":1941,"_60":1942},{},[1943],"Now, this is where we start thinking about how to solve these issues. Everything described so far can break, and will continue to break even if I fix it:",{"_300":301,"_54":576,"_303":1945,"_60":1946},{"_62":121},[1947],["SingleFetchClassInstance",1948],{"_300":301,"_54":348,"_303":1949,"_60":1950},{"_351":1951},[122],"#the-watchdog-script",{"_300":301,"_54":302,"_303":1953,"_60":1954},{},[1955],["SingleFetchClassInstance",1956],{"_300":301,"_54":420,"_303":1957,"_60":1958},{},[1959,1960,1961],["SingleFetchClassInstance",1971],["SingleFetchClassInstance",1967],["SingleFetchClassInstance",1962],{"_300":301,"_54":427,"_303":1963,"_60":1964},{"_430":1965,"_432":675,"_434":435,"_436":1966,"_438":439},[],"Manual Postgres cluster diagram","https://planetscale-images.imgix.net/assets/part1-manual-cluster-C6iQCdoL.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1968,"_60":1969},{"_444":1970,"_446":447},[],"https://planetscale-images.imgix.net/assets/part1-manual-cluster-darkmode-VBZiV-Rl.png?auto=compress%2Cformat",{"_300":301,"_54":441,"_303":1972,"_60":1973},{"_444":1966,"_446":451},[],{"_300":301,"_54":302,"_303":1975,"_60":1976},{},[1977],"But we still have a problem: the first time the primary is recreated with a different IP, the whole cluster falls apart.",{"_300":301,"_54":550,"_303":1979,"_60":1980},{"_553":1981,"_555":1982},[],"

# on each replica's postgresql.auto.conf, until the primary is recreated with a new IP\nprimary_conninfo = 'host=10.4.7.21 port=5432 user=replicator ...'\n

","conf",{"_300":301,"_54":302,"_303":1984,"_60":1985},{},[1986,1987,1988,1989,1990,1991,1992],"The first thing I do is hard-code the IPs. I write ",["SingleFetchClassInstance",2000],"'s address into the replicas' config, I list the replicas' addresses in the primary's ",["SingleFetchClassInstance",1996],", and I keep a small ",["SingleFetchClassInstance",1993]," table and save it somewhere.",{"_300":301,"_54":402,"_303":1994,"_60":1995},{},[1482],{"_300":301,"_54":402,"_303":1997,"_60":1998},{},[1999],"pg_hba.conf",{"_300":301,"_54":402,"_303":2001,"_60":2002},{},[1556],{"_300":301,"_54":302,"_303":2004,"_60":2005},{},[2006],"Here is another thing we have to solve. The replicas need to reach the primary, and the primary needs to accept their connections. And every one of these addresses is an IP that changes when a container restarts.",{"_300":301,"_54":576,"_303":2008,"_60":2009},{"_62":124},[2010],["SingleFetchClassInstance",2011],{"_300":301,"_54":348,"_303":2012,"_60":2013},{"_351":2014},[125],"#they-have-to-find-each-other",{"_300":301,"_54":302,"_303":2016,"_60":2017},{},[2018,2019,2020],"Now we have three nodes with three Postgres instances. One of the instances is the primary (here it's ",["SingleFetchClassInstance",2021],"). But this raises new problems, like what to do if the primary's node dies?",{"_300":301,"_54":402,"_303":2022,"_60":2023},{},[1556],{"_300":301,"_54":302,"_303":2025,"_60":2026},{},[2027,2028,1724,2029,2030,2031,2032,2033,2034],"A single Postgres instance is a single point of failure. We want high availability: one primary and two replicas. These need to be on three different machines, with streaming replication between them. So we do the same steps again, three times, on ",["SingleFetchClassInstance",2047],["SingleFetchClassInstance",2043],", and ",["SingleFetchClassInstance",2039],". I also wire up replication by hand: ",["SingleFetchClassInstance",2035],", replication slots, all of it.",{"_300":301,"_54":402,"_303":2036,"_60":2037},{},[2038],"primary_conninfo",{"_300":301,"_54":402,"_303":2040,"_60":2041},{},[2042],"node-19",{"_300":301,"_54":402,"_303":2044,"_60":2045},{},[2046],"node-12",{"_300":301,"_54":402,"_303":2048,"_60":2049},{},[1556],{"_300":301,"_54":576,"_303":2051,"_60":2052},{"_62":127},[2053],["SingleFetchClassInstance",2054],{"_300":301,"_54":348,"_303":2055,"_60":2056},{"_351":2057},[128],"#one-isnt-enough",{"_300":301,"_54":302,"_303":2059,"_60":2060},{},[2061],"These are a lot of steps, and each one can fail halfway. And if the disk fills up later, Postgres stops accepting writes and we have to resize the volume by hand: first through the cloud provider, then again inside the filesystem.",{"_300":301,"_54":550,"_303":2063,"_60":2064},{"_553":2065,"_555":1851},[],"

# provision + attach first with cloud CLI, then on the node:\nmkfs.ext4 /dev/nvme1n1\nmkdir -p /var/lib/pg-data\nmount /dev/nvme1n1 /var/lib/pg-data\ndocker run -d --name pg \\\n  -e POSTGRES_PASSWORD=secret \\\n  -v /var/lib/pg-data:/var/lib/postgresql \\\n  postgres:18\n

",{"_300":301,"_54":302,"_303":2067,"_60":2068},{},[2069],"Container storage is ephemeral, so I have to attach a real block device. In the cloud this is an EBS volume (e.g. on AWS); on bare metal it's a physical disk. Assuming it's a block device, this is what we usually do: provision the volume, attach it to the node, format it, mount it, and point Postgres' data directory at the mount.",{"_300":301,"_54":576,"_303":2071,"_60":2072},{"_62":130},[2073],["SingleFetchClassInstance",2074],{"_300":301,"_54":348,"_303":2075,"_60":2076},{"_351":2077},[131],"#it-needs-a-real-disk",{"_300":301,"_54":302,"_303":2079,"_60":2080},{},[2081,2082,2083],"I picked ",["SingleFetchClassInstance",2084]," because it looked idle enough. I start keeping track of it, save it in some sort of config file, and push it to some repo.",{"_300":301,"_54":402,"_303":2085,"_60":2086},{},[1556],{"_300":301,"_54":550,"_303":2088,"_60":2089},{"_553":2090,"_555":1851},[],"

ssh node-07 'docker run -d --name pg ... postgres:18'\n

",{"_300":301,"_54":302,"_303":2092,"_60":2093},{},[2094,2095,2096,2097,2098],"Imagine we have hundreds of nodes (servers) we can use. I already have other workloads running on them. I need to decide ",["SingleFetchClassInstance",2102]," runs this database. So I ",["SingleFetchClassInstance",2099]," into the box that looks the least busy and start the container there.",{"_300":301,"_54":402,"_303":2100,"_60":2101},{},[601],{"_300":301,"_54":368,"_303":2103,"_60":2104},{},[2105],"which one",{"_300":301,"_54":576,"_303":2107,"_60":2108},{"_62":133},[2109],["SingleFetchClassInstance",2110],{"_300":301,"_54":348,"_303":2111,"_60":2112},{"_351":2113},[134],"#pick-a-node-by-hand",{"_300":301,"_54":302,"_303":2115,"_60":2116},{},[2117,2118,2119,2120,2121],"There is already a gap between what I ",["SingleFetchClassInstance",2125]," (Postgres, running, with my data) and what I ",["SingleFetchClassInstance",2122]," (a container whose storage disappears when the container or node goes away). The rest of this post is about that gap and the machinery we build to close it.",{"_300":301,"_54":368,"_303":2123,"_60":2124},{},[1842],{"_300":301,"_54":368,"_303":2126,"_60":2127},{},[1846],{"_300":301,"_54":302,"_303":2129,"_60":2130},{},[2131],"That's it. Postgres is running. My app connects to it, writes some rows, and everything works fine. But then the machine goes away: the cloud provider reclaims the instance (hardware fails, or a spot instance gets taken back), or I ship a new version of my setup, which means stopping the old container and starting a fresh one in its place. Either way, the container is replaced, and my data is gone. The container storage was ephemeral, and I did not attach any persistent volume to it.",{"_300":301,"_54":550,"_303":2133,"_60":2134},{"_553":2135,"_555":1851},[],"

docker run -d --name pg \\\n  -e POSTGRES_PASSWORD=secret \\\n  postgres:18\n

",{"_300":301,"_54":302,"_303":2137,"_60":2138},{},[2139],"Let's start from scratch. I want to run Postgres on a Linux box, and I need it inside a container. To start it, we run:",{"_300":301,"_54":576,"_303":2141,"_60":2142},{"_62":136},[2143],["SingleFetchClassInstance",2144],{"_300":301,"_54":348,"_303":2145,"_60":2146},{"_351":2147},[137],"#one-container-one-machine",{"_300":301,"_54":343,"_303":2149,"_60":2150},{"_62":104},[2151],["SingleFetchClassInstance",2152],{"_300":301,"_54":348,"_303":2153,"_60":2154},{"_351":2155},[105],"#part-1-running-postgres-by-hand",{"_300":301,"_54":354,"_303":2157,"_60":2158},{},[],{"_300":301,"_54":2160,"_303":2161,"_60":2162},"Callout",{"_2194":2195},[2163,2164],["SingleFetchClassInstance",2169],["SingleFetchClassInstance",2165],{"_300":301,"_54":302,"_303":2166,"_60":2167},{},[2168],"We're going to start slow and gradually ramp things up. Each part builds on the previous.",{"_300":301,"_54":302,"_303":2170,"_60":2171},{},[2172,2173,2174,2175,1724,2176,2030,2177,2178],"A working understanding of containers and ",["SingleFetchClassInstance",2190]," helps, but you don't need to be a Kubernetes expert. I'll use terms like ",["SingleFetchClassInstance",2187],["SingleFetchClassInstance",2183],["SingleFetchClassInstance",2179],", and introduce the parts that matter as we go.",{"_300":301,"_54":368,"_303":2180,"_60":2181},{},[2182],"eventual consistency",{"_300":301,"_54":368,"_303":2184,"_60":2185},{},[2186],"fan-in",{"_300":301,"_54":368,"_303":2188,"_60":2189},{},[1898],{"_300":301,"_54":402,"_303":2191,"_60":2192},{},[2193],"kubectl","type","note",{"_300":301,"_54":302,"_303":2197,"_60":2198},{},[2199],"Before we look at a single line of Kubernetes, we're going to run a production database by hand and slowly let the feedback loop appear on its own. Then we'll map that loop to Kubernetes, with the pieces production needs: a store, watches, queues, retries, and more. At the end, we'll look at what one of these loops looks like in a real operator.",{"_300":301,"_54":302,"_303":2201,"_60":2202},{},[2203],"An operator is a feedback controller. It's the same closed loop that runs a thermostat or keeps your car at a fixed speed on cruise control. In our case, the thing being controlled is a database. I have been building these loops for years, and the best way I know to make them click is to ignore Kubernetes at the beginning. Kubernetes is full of control theory, even if we don't call it that in the day-to-day.",{"_300":301,"_54":302,"_303":2205,"_60":2206},{},[2207],"People ask me what an operator actually does. The canonical answer is: \"it reconciles desired state.\" This is correct, but it also tells you almost nothing.",{"_300":301,"_54":302,"_303":2209,"_60":2210},{},[2211],"For the last decade, Kubernetes has been the backdrop to most of my work: operating clusters, helping build hosted Kubernetes, and writing Kubernetes operators. At PlanetScale, that now means running stateful systems like Postgres and MySQL in production. Kubernetes has many faces, but here I want to talk about one face only: why it is so good at running workloads at scale.","current",{"_2214":19,"_2215":2216,"_2217":19},"development","env",{"_2218":2219,"_2220":2221,"_2222":2223,"_2224":2225,"_2226":2227},"userSignedIn","IMAGE_CDN","https://planetscale-images.imgix.net","IMAGE_CDN_ENABLED","true","INTERNAL_API","https://api.planetscale.com","RELEASE","5bd7300dff90db3f34bb196df51e0222cbf1da89","SENTRY_DSN","https://bd81903b44804e22a06bdc0c1a91b303@o499952.ingest.us.sentry.io/4504531942572032"]

Posts by Fatih Arslan

The feedback loops behind Kubernetes