Kubernetes Architecture - Control Plane & Nodes

While Docker solved the packaging and runtime isolation challenges of single-node containers, enterprise applications operating at global scale quickly outgrew single-machine boundaries. Running a modern platform demands managing hundreds or thousands of container instances distributed across clusters of bare-metal and cloud virtual machines.

At this scale, manual container management becomes impossible: What happens when an underlying physical host experiences a hardware failure? How are rolling zero-downtime updates coordinated without dropping live customer traffic? How are compute workloads automatically placed onto nodes with adequate CPU and memory capacity?

Originally designed by Google engineers based on over a decade of production experience with internal cluster managers Borg and Omega, Kubernetes (K8s) was open-sourced in 2014 to serve as the definitive container orchestration platform.

At its core, Kubernetes is a distributed declarative state machine. This module delivers an exhaustive, university-grade architectural analysis of the Kubernetes control plane components (kube-apiserver, etcd, kube-scheduler, kube-controller-manager), worker node agents (kubelet, kube-proxy, CRI, CNI), and the mathematical mechanics of declarative reconciliation loops and Raft consensus quorum.

Global Cluster Topology: Control Plane vs Worker Nodes#

A Kubernetes cluster is fundamentally partitioned into two distinct operational planes: the Control Plane (the administrative brain) and Worker Nodes (the execution muscle).

+---------------------------------------------------------------------------------------------------+
| Kubernetes Cluster Architecture                                                                   |
|                                                                                                   |
|  CONTROL PLANE (MASTER NODES)                                                                     |
|                                                                                                   |
|  +---------------------------------------------------------------------------------------------+  |
|  | kube-apiserver (Stateless Ingress Gateway)                                                  |  |
|  | - Authentication, RBAC Authorization, Admission Control (Mutating/Validating), API Schema   |  |
|  +------------------------------+-------------------------------+------------------------------+  |
|                                 |                               |                                 |
|                 +---------------+---------------+               | Raft Consensus                  |
|                 |                               |               v                                 |
|  +--------------v---------------+  +------------v---------------+  +---------------------------+  |
|  | kube-scheduler               |  | kube-controller-manager    |  | etcd Cluster              |  |
|  | - 2-Phase Placement:         |  | - Core Reconciliation Loops|  | - Distributed Consistent  |  |
|  |   1. Filtering (Predicates)  |  |   (Deployment, Node,       |  |   Key-Value Store          |  |
|  |   2. Scoring (Priorities)    |  |    EndpointSlice, Service) |  | - Quorum: 2F + 1 Nodes    |  |
|  +------------------------------+  +----------------------------+  +---------------------------+  |
|                                 |                                                                 |
+---------------------------------+-----------------------------------------------------------------+
                                  | mTLS Communication (Port 6443 / 10250)
        +-------------------------+-------------------------+
        |                                                   |
        v                                                   v
+---------------------------------------+   +---------------------------------------+
| WORKER NODE 1                         |   | WORKER NODE 2                         |
|                                       |   |                                       |
|  +---------------------------------+  |   |  +---------------------------------+  |
|  | kubelet                         |  |   |  | kubelet                         |  |
|  | - Node Agent, Pod Lifecycle     |  |   |  | - Node Agent, Pod Lifecycle     |  |
|  | - Interacts via CRI (containerd)|  |   |  | - Interacts via CRI (containerd)|  |
|  +----------------+----------------+  |   |  +----------------+----------------+  |
|                   |                   |   |                   |                   |
|  +----------------v----------------+  |   |  +----------------v----------------+  |
|  | kube-proxy                      |  |   |  | kube-proxy                      |  |
|  | - Service VIP Routing           |  |   |  | - Service VIP Routing           |  |
|  | - IPVS / iptables rules         |  |   |  | - IPVS / iptables rules         |  |
|  +---------------------------------+  |   |  +---------------------------------+  |
|                                       |   |                                       |
|  +---------------------------------+  |   |  +---------------------------------+  |
|  | Container Runtime: containerd   |  |   |  | Container Runtime: containerd   |  |
|  | Pod A (App)     Pod B (Sidecar) |  |   |  | Pod C (Worker)  Pod D (App)     |  |
|  +---------------------------------+  |   |  +---------------------------------+  |
|                                       |   |                                       |
|  +---------------------------------+  |   |  +---------------------------------+  |
|  | CNI Plugin (Calico/Cilium-eBPF) |  |   |  | CNI Plugin (Calico/Cilium-eBPF) |  |
|  | Flat Pod-to-Pod Overlay Network |  |   |  | Flat Pod-to-Pod Overlay Network |  |
|  +---------------------------------+  |   |  +---------------------------------+  |
+---------------------------------------+   +---------------------------------------+

The Control Plane Components#

The control plane makes global decisions about the cluster (such as scheduling workloads, detecting node failures, and maintaining desired replica counts).

1. `kube-apiserver`: The Central Gateway#

The API server is the central hub of the entire Kubernetes cluster. It exposes the Kubernetes HTTP/JSON REST API (standard port: 6443).

• Statelessness: kube-apiserver stores zero state in memory or on local disk. It can be scaled horizontally behind a Layer 4 load balancer (HAProxy or cloud network load balancer).
• The Exclusive Gateway to `etcd`: No component in the entire cluster—not even the scheduler or controller manager—is permitted to communicate directly with `etcd`. Every read and write to cluster state must pass through kube-apiserver.

The Request Admission Pipeline:#

When a client (e.g., kubectl apply -f deployment.yaml) sends a request to the API server, it traverses a four-stage pipeline:

1. Authentication: Verifies caller identity using TLS client certificates, OpenID Connect (OIDC) JWT tokens, or Webhook authenticators.
2. Authorization: Evaluates whether the authenticated identity has permission to perform the requested verb (get, list, create, update, delete) on the target resource using Role-Based Access Control (RBAC).
3. Mutating Admission Webhooks: External webhooks that can modify the incoming resource definition before storage (e.g., automatically injecting Istio sidecar proxy containers or adding default security context constraints).
4. Schema Validation & Validating Admission Webhooks: Verifies that the manifest conforms to the OpenAPI specification and executes final business policy validation (e.g., rejecting pods that do not define CPU limits).
5. Persistence: The validated resource is written atomically to etcd.

2. `etcd`: The Distributed Consistent Key-Value Store#

etcd is an open-source, strongly consistent, distributed key-value store created by CoreOS. It holds the complete, canonical declarative state of the Kubernetes cluster.

• Consensus via Raft: etcd uses the Raft consensus algorithm to replicate state across a cluster of nodes. Raft guarantees that all nodes agree on the exact sequence of state transitions, ensuring strong consistency (linearizable reads and writes).
• The Quorum Invariant: To tolerate F node failures, an etcd cluster must contain at least:

N = 2F + 1 nodesA 3-node cluster tolerates 1 failure; a 5-node cluster tolerates 2 failures. An even number of nodes (e.g., 4) provides zero additional fault tolerance over 3 nodes (both tolerate only 1 failure) while increasing network consensus communication overhead.

• Watch API: etcd provides an efficient streaming Watch API using HTTP/2 streams. Rather than polling for changes, the API server "watches" key prefixes (e.g., /registry/pods/). When a key changes, etcd immediately streams the revision event to the API server.

3. `kube-scheduler`: The Workload Placement Engine#

The scheduler monitors kube-apiserver for newly created pods that have not yet been assigned to a worker node (spec.nodeName == "").

The scheduler operates via a Two-Phase Algorithm:

                 [ Unscheduled Pod Arrives ]
                              |
                              v
       +----------------------------------------------+
       | Phase 1: Filtering (Predicates)              |
       | - Discard nodes with insufficient CPU/RAM    |
       | - Discard nodes violating NodeSelector       |
       | - Discard nodes with un-tolerated Taints     |
       | - Discard nodes with hostPort conflicts      |
       +----------------------+-----------------------+
                              |
                              v (Remaining Feasible Nodes)
       +----------------------------------------------+
       | Phase 2: Scoring (Priorities)                |
       | - Rank nodes using weighted scoring functions|
       | - NodeResourcesBalancedAllocation            |
       | - ImageLocalityPriority (Image cached?)      |
       | - PodTopologySpread / Anti-Affinity          |
       +----------------------+-----------------------+
                              |
                              v (Select Highest Scoring Node)
       [ Dispatch Bind Request to API Server: spec.nodeName = "node-3" ]

4. `kube-controller-manager`: The Core Control Loops#

The controller manager executes the core control loops of Kubernetes. A controller is an infinite loop that continuously reconciles the current observed state of the cluster with the desired state recorded in etcd.

Key controllers packaged within the single binary:

• Deployment Controller: Manages ReplicaSets to match desired rolling deployment configurations.
• Node Controller: Monitors worker node heartbeats. If a node fails to report for pod-eviction-timeout (default: 5 minutes), the node controller marks the node Unhealthy and evicts its pods to healthy nodes.
• EndpointSlice Controller: Keeps track of running pod IP addresses and dynamically updates Service endpoint slices.

The Worker Node Architecture#

Worker nodes host the actual user application containers and provide the runtime compute environment.

1. `kubelet`: The Primary Node Agent#

The kubelet is an agent that runs on every node in the cluster. It does not manage containers directly; rather, it coordinates with the container runtime.

• Pod Lifecycle Management: The kubelet watches the API server for PodSpecifications assigned to its node. It instructs the local container runtime to pull images, create container namespaces, and start processes.
• Health Probes: Executes livenessProbe, readinessProbe, and startupProbe definitions (executing shell commands, HTTP GET requests, or TCP socket checks).
• CRI Interface: Interacts with container runtimes via the standard Container Runtime Interface (CRI) gRPC protocol.
• Node Eviction: Monitors node memory and disk thresholds. If /var/lib/kubelet or /var/lib/containerd runs out of space, kubelet proactively terminates pods based on Quality of Service (QoS) classes.

2. `kube-proxy`: Service VIP Networking#

Kubernetes Pods are ephemeral; they are created and destroyed dynamically, and their IP addresses change frequently. To provide a stable network endpoint, Kubernetes defines the Service abstraction, which assigns a persistent Virtual IP (ClusterIP).

kube-proxy runs on every node to program the host kernel networking rules so that traffic sent to the Service Virtual IP is load-balanced to the actual pod IPs.

Modes of `kube-proxy`:#

1. iptables Mode:

kube-proxy generates Netfilter iptables chains for every service and endpoint.

The O(N) Degradation Problem*: In large clusters with 10,000 services and 50,000 endpoints, iptables rules grow to hundreds of thousands of sequential lines. Because Linux Netfilter evaluates iptables sequentially (O(N)), packet traversal latency degrades, and updating a single service requires rebuilding the entire table in kernel memory.

1. IPVS Mode (IP Virtual Server):

Modern production clusters utilize IPVS, a Layer 4 transport load balancer built directly into the Linux kernel.

O(1) Hash Table Lookup*: IPVS stores connection routing rules in high-performance kernel hash tables (O(1) complexity). Performance remains completely flat whether managing 10 services or 100,000 services.

* Supports advanced load-balancing algorithms: Least-Connection, Weighted Round-Robin, Locality-Based.

The Container Network Interface (CNI)#

Kubernetes enforces a strict, flat networking model across the entire cluster:

1. Every Pod receives its own unique IP address.
2. All Pods can communicate with all other Pods across nodes without Network Address Translation (NAT).
3. Agents on a node (like kubelet) can communicate with all Pods on that same node.

The implementation of this network is delegated to CNI Plugins:

• Calico / Flannel: Establishes Virtual Extensible LAN (VXLAN) or BGP (Border Gateway Protocol) overlay networks that encapsulate cross-node pod traffic.
• Cilium: The cutting-edge industry standard. Utilizes eBPF (Extended Berkeley Packet Filter) directly inside the Linux kernel to bypass iptables completely, achieving near bare-metal socket-to-socket throughput and hardware-level security observability.

Checkpoint Practice Problems#

Test your architectural depth in distributed systems consensus, Kubernetes scheduling mathematics, and control plane engineering through these comprehensive problems.

Problem 1: Raft Consensus Quorum & etcd Node Fault Tolerance Analysis#

An infrastructure architect designs an etcd cluster to support a high-availability Kubernetes control plane deployed across multi-region cloud availability zones.

The Raft consensus algorithm dictates that an etcd cluster of N nodes requires a majority quorum of nodes to commit state mutations:

Q = ≤ft⌊ (N) / (2) ⌋ + 1The fault tolerance (F) is the maximum number of node failures the cluster can sustain while still maintaining quorum:

F = N - Q = N - ≤ft(≤ft⌊ (N) / (2) ⌋ + 1)Evaluate cluster configurations for N = 1, 2, 3, 4, 5, 6, 7:

1. Calculate the Quorum size (Q) and Fault Tolerance (F) for each configuration.
2. Formally prove why deploying an even number of etcd nodes (e.g., N = 4 or N = 6) is an operational anti-pattern compared to odd configurations (N = 3 or N = 5).
3. In a 3-node etcd cluster deployed across three Availability Zones (AZ-1, AZ-2, AZ-3), AZ-1 experiences a complete fiber cut (network partition separating AZ-1 from AZ-2 and AZ-3). Detail the exact sequence of leader election and write commit events that occur on both sides of the partition.

Solution:

Step 1: Quorum and Fault Tolerance Table:

Step 2: Mathematical Proof of the Even-Node Anti-Pattern:

Consider moving from N = 3 to N = 4:

• For N = 3: Quorum Q = 2. Fault tolerance F = 3 - 2 = 1. The cluster can survive 1 dead node.
• For N = 4: Quorum Q = 3. Fault tolerance F = 4 - 3 = 1. The cluster can survive 1 dead node.

Notice that adding a 4th node confers ZERO additional fault tolerance!

Furthermore, it actively harms system resilience:

1. Higher Failure Probability: With 4 physical nodes, the statistical probability that any individual node or network link fails is higher than with 3 nodes.
2. Quorum Fragility: In a 4-node cluster, losing 2 nodes destroys quorum (4 - 2 = 2 < 3). In a 3-node cluster, losing 2 nodes also destroys quorum. However, if a network partition splits the 4-node cluster evenly (2 nodes ≤ftrightarrow 2 nodes), neither side has quorum (2 < 3). The entire cluster freezes and accepts zero writes! In contrast, a 3-node cluster split across a partition naturally divides into 2 ≤ftrightarrow 1, allowing the majority side (2 ≥ 2) to continue servicing traffic uninterrupted.
3. RPC Overhead: Raft requires the leader to broadcast heartbeat AppendEntries RPCs to all peers. A 4th node adds network serialization overhead without improving availability.

Step 3: Network Partition Event Walkthrough:

• Initial State: Nodes A (AZ-1), B (AZ-2), C (AZ-3). Node A is the Raft Leader.
• Partition occurs: AZ-1 is isolated from AZ-2 and AZ-3.
• On Partition Side 1 (Node A alone):

* Node A receives a client write from kube-apiserver.

* Node A appends the entry to its local log and attempts to replicate to B and C.

* RPCs time out. Node A receives 0 responses.

* Number of nodes agreeing = 1 (Node A only).

* Quorum requirement = 2.

Node A cannot commit the write*. The request blocks or fails.

• On Partition Side 2 (Nodes B and C):

* Nodes B and C stop receiving leader heartbeats from A.

* Election timeout elapses (e.g., 300 ms).

* Node B increments term, becomes Candidate, and requests votes from C.

* Node C votes for B.

* Node B receives 2 votes out of 3. Majority quorum achieved (2 ≥ 2)!

* Node B becomes the new Raft Leader.

* kube-apiserver instances in AZ-2 and AZ-3 route writes to Node B. Node B replicates to C, commits the entries, and successfully mutates state.

• Partition Resolution:

* When the fiber link heals, Node A detects Node B's higher term number, steps down to Follower, and updates its log from B. Zero data is lost, and linearizable consistency is maintained.

Problem 2: Kubernetes Scheduler Scoring Matrix Algorithm#

An unscheduled pod requiring:

CPU Request = 1,000 mCPU = 1.0 vCPUMemory Request = 2,048 MiB = 2.0 GiBis processed by kube-scheduler.

During Phase 1 (Filtering), all three candidate nodes (Node 1, Node 2, Node 3) pass all predicate filters.

During Phase 2 (Scoring), the scheduler evaluates the nodes using two standard priority functions:

1. LeastRequestedPriority (S_{least}): Favors nodes with fewer allocated resources (spreads workloads across the cluster):

S_{least} = (≤ft((Capacity_{cpu} - TotalAllocated_{cpu}) / (Capacity_{cpu}) × 10) + ≤ft((Capacity_{mem} - TotalAllocated_{mem}) / (Capacity_{mem}) × 10)) / (2)2. ImageLocalityPriority (S_{image}): Rewards nodes that already have the container image cached locally in their CRI storage:

* Node with image cached: Score = 10

* Node without image cached: Score = 0

Priority Weights:

W_{least} = 1.0, W_{image} = 2.0Total Score = (S_{least} × W_{least}) + (S_{image} × W_{image})Node Metrics:

• Node 1:

* Capacity: 8.0 vCPU, 16.0 GiB RAM.

* Current Allocated (including new pod): 5.0 vCPU, 10.0 GiB RAM.

Image Cached? YES*.

• Node 2:

* Capacity: 4.0 vCPU, 8.0 GiB RAM.

* Current Allocated (including new pod): 2.0 vCPU, 4.0 GiB RAM.

Image Cached? NO*.

• Node 3:

* Capacity: 16.0 vCPU, 32.0 GiB RAM.

* Current Allocated (including new pod): 4.0 vCPU, 8.0 GiB RAM.

Image Cached? NO*.

Calculate:

1. The S_{least} score for Node 1, Node 2, and Node 3.
2. The final weighted Total Score for all three nodes.
3. Identify which node the scheduler selects to bind the pod.

Solution:

Step 1: Calculate S_{least} for each node:

• Node 1:

Free CPU fraction = (8.0 - 5.0) / 8.0 = 3.0 / 8.0 = 0.375 → 3.75 pts.

Free Mem fraction = (16.0 - 10.0) / 16.0 = 6.0 / 16.0 = 0.375 → 3.75 pts.

S_{least}(Node 1) = (3.75 + 3.75) / (2) = 3.75 Node 2*:

Free CPU fraction = (4.0 - 2.0) / 4.0 = 2.0 / 4.0 = 0.500 → 5.00 pts.

Free Mem fraction = (8.0 - 4.0) / 8.0 = 4.0 / 8.0 = 0.500 → 5.00 pts.

S_{least}(Node 2) = (5.00 + 5.00) / (2) = 5.00 Node 3*:

Free CPU fraction = (16.0 - 4.0) / 16.0 = 12.0 / 16.0 = 0.750 → 7.50 pts.

Free Mem fraction = (32.0 - 8.0) / 32.0 = 24.0 / 32.0 = 0.750 → 7.50 pts.

S_{least}(Node 3) = (7.50 + 7.50) / (2) = 7.50Step 2: Calculate Weighted Total Score:

• Node 1:

S_{image} = 10 (Image cached).

Total Score(Node 1) = (3.75 × 1.0) + (10.0 × 2.0) = 3.75 + 20.0 = 23.75 Node 2*:

S_{image} = 0 (Image not cached).

Total Score(Node 2) = (5.00 × 1.0) + (0.0 × 2.0) = 5.00 + 0.0 = 5.00 Node 3*:

S_{image} = 0 (Image not cached).

Total Score(Node 3) = (7.50 × 1.0) + (0.0 × 2.0) = 7.50 + 0.0 = 7.50Step 3: Scheduling Selection:

Even though Node 3 had vastly more unallocated capacity (S_{least} = 7.50), the heavy weight assigned to ImageLocalityPriority (W_{image} = 2.0) made caching the dominant factor.

The scheduler selects Node 1 (Score = 23.75) to eliminate image pull network transfer time, optimizing pod startup velocity.

Problem 3: `kube-proxy` iptables vs IPVS Rule Complexity Analysis#

An enterprise Kubernetes cluster hosts S = 5,000 Services. Each service maps to an average of E = 10 Pod Endpoints.

Total endpoints in cluster = 5,000 × 10 = 50,000 active endpoints.

Under iptables mode:

• Each service creates a dedicated KUBE-SVC-* chain containing E sequential probability-weighted rules:

Rule 1 matches with probability 1/10.

Rule 2 matches with probability 1/9.

...

Rule 10 matches with probability 1.0.

• The host kernel must traverse rules sequentially: Average comparisons to match a service = S / 2 = 2,500.
• Average comparisons within the service chain to match an endpoint = E / 2 = 5.
• Total average sequential rule comparisons per incoming packet = 2,500 + 5 = 2,505 rule checks.
• Empirically, each sequential iptables check consumes 0.12 microseconds of kernel CPU execution.

Under IPVS mode:

• Services are indexed inside an in-memory IPVS kernel hash table.
• The kernel computes hash(Destination VIP, Port) in O(1) constant time (0.40 microseconds flat).
• Endpoint selection executes via a hardware-accelerated round-robin pointer dereference (0.10 microseconds).
• Total IPVS processing time = 0.50 microseconds flat.

1. Calculate the kernel processing latency per network packet under iptables mode versus IPVS mode.
2. If the worker node processes an incoming throughput of 100,000 packets/second, compute the total CPU core time consumed strictly by routing rule lookups for both modes.
3. Compute the performance acceleration factor of IPVS over iptables.

Solution:

Step 1: Calculate Packet Processing Latency:

• iptables Mode:

Latency_{iptables} = 2,505 comparisons × 0.12 μs/comp = 300.6 microseconds IPVS Mode*:

Latency_{ipvs} = 0.50 microsecondsStep 2: Compute CPU Core Consumption at 100,000 Packets/sec:

• iptables Mode:

Total CPU time per second:

CPU Time = 100,000 packets/sec × 300.6 × 10^{-6} sec/packet = 30.06 core-seconds/secThis means iptables rule evaluation requires 30 full CPU cores dedicated strictly to network packet filtering! A standard 16-core or 32-core worker node is completely overwhelmed by kernel softirq processing, dropping packets and stalling user applications.

• IPVS Mode:

Total CPU time per second:

CPU Time = 100,000 packets/sec × 0.50 × 10^{-6} sec/packet = 0.05 core-seconds/secIPVS consumes only 0.05 of a single CPU core (5% of 1 core) to process the exact same 100,000 packets/sec!

Step 3: Acceleration Factor:

Speedup = (300.6 μs) / (0.50 μs) ≈ 601.2×IPVS accelerates service packet routing by over 600×, proving why enterprise Kubernetes clusters operating above a few hundred services must switch kube-proxy to IPVS mode.

Problem 4: Multi-Master High-Availability Control Plane Topology#

Design a production-grade, highly available, multi-master Kubernetes control plane architecture that eliminates all single points of failure (SPOF) across physical hardware and network boundaries.

Your architecture must detail:

1. The topology of the kube-apiserver instances and their Layer 4 load balancer.
2. The stacked etcd topology across three distinct Availability Zones.
3. How kube-scheduler and kube-controller-manager manage leader election when multiple instances run simultaneously.

Solution:

+-----------------------------------------------------------------------------------+
| Production High-Availability Control Plane Topology                               |
|                                                                                   |
|                   [ External Traffic / Worker Node Traffic ]                      |
|                                        |                                          |
|                                        v                                          |
|  +-----------------------------------------------------------------------------+  |
|  | Layer 4 Highly Available Load Balancer (HAProxy + Keepalived / AWS NLB)     |  |
|  | Virtual IP: 10.0.0.100:6443 (Active-Passive VRRP or Anycast BGP)            |  |
|  +------------------+------------------+------------------+--------------------+  |
|                     |                  |                  |                       |
|         +-----------+                  |                  +-----------+           |
|         |                              |                              |           |
|         v (AZ-1)                       v (AZ-2)                       v (AZ-3)    |
|  +--------------------+         +--------------------+         +--------------------+
|  | Master Node 1      |         | Master Node 2      |         | Master Node 3      |
|  |                    |         |                    |         |                    |
|  | - kube-apiserver   |         | - kube-apiserver   |         | - kube-apiserver   |
|  | - kube-controller  |         | - kube-controller  |         | - kube-controller  |
|  |   (Leader: Active) |         |   (Standby)        |         |   (Standby)        |
|  | - kube-scheduler   |         | - kube-scheduler   |         | - kube-scheduler   |
|  |   (Standby)        |         |   (Leader: Active) |         |   (Standby)        |
|  | - etcd Node 1      |         | - etcd Node 2      |         | - etcd Node 3      |
|  +---------+----------+         +---------+----------+         +---------+----------+
|            |                              |                              |        |
|            +<============================>+<============================>+        |
|                  mTLS Raft Consensus Replication (Port 2379 / 2380)               |
+-----------------------------------------------------------------------------------+

Architectural Key Pillars:#

1. Stateless API Load Balancing:

The three kube-apiserver instances are identical and stateless. Worker nodes and operators communicate strictly via the Virtual IP 10.0.0.100:6443. The load balancer health-checks /livez on each API server, automatically bypassing any crashed master node.

1. Stacked etcd Quorum:

Each master node co-locates an etcd member. The three members communicate over mutual TLS (mTLS) using ports 2379 (client API) and 2380 (peer Raft communication). Distributed across three independent availability zones, the cluster tolerates the catastrophic loss of an entire cloud datacenter without service interruption (3 - 2 = 1 zone loss tolerated).

1. Active-Passive Leader Election for Schedulers & Controllers:

Unlike kube-apiserver (which runs active-active), the cluster cannot have multiple schedulers or controllers executing mutations simultaneously; doing so would cause conflicting scheduling assignments and duplicate pod spawns.

`kube-scheduler` and `kube-controller-manager` implement Lease-based Leader Election* using Kubernetes coordination.k8s.io/v1 Lease resources.

* All three instances run concurrently, but only one acquires the lock (Leader).

* The leader periodically renews its lease (leaseDurationSeconds: 15).

* If Master Node 1 crashes, the standby instances on Master Node 2 and Node 3 detect lease expiration and immediately elect a new leader, guaranteeing seamless high availability.