Integrate service mesh with Kubernetes endpoints

Introduction

When you register a service in a platform's control plane and then watch a rolling deployment shift its pods across nodes, you quickly learn that a hand-maintained address list is stale the moment you write it. Service meshes route traffic by resolving logical service names to concrete pod IPs, but the mesh's picture of the world is only as fresh as the source feeding it. On Kubernetes, the authoritative source is the pair of Service and Endpoints objects the API server maintains — yet most platform registries are populated by humans or CI scripts that never see a pod reschedule. The gap between "what the registry believes" and "what Kubernetes actually runs" is where dropped requests, black-hole routes, and 30-second outages live. By the end of this lesson, you'll be able to connect a platform service registry to Kubernetes Service and Endpoints resources using a label-based watch loop, so that mesh routing tables reconcile automatically as pods come and go.

Key Terminology

• Service Registry: The platform's authoritative record of every routable service, its logical name, and its current backend addresses. The mesh's sidecars read from this registry to build their routing tables.
• Endpoints Object: A Kubernetes resource, one per Service, that lists the concrete pod IP/port pairs currently backing that service. The kube-controller-manager rewrites it every time a matching pod becomes ready or unready.
• Label Selector: A key/value query (for example mesh-managed=true) that matches a subset of Kubernetes objects. Label-based discovery means the registry syncs only the services a team explicitly opts in, not every object in the cluster.
• Watch Stream: A long-lived HTTP connection to the Kubernetes API server that pushes ADDED, MODIFIED, and DELETED events as objects change, replacing wasteful full-list polling with incremental deltas.
• Resource Version: A monotonic token on every Kubernetes object that marks its position in the change stream. Resuming a watch from the last seen resourceVersion lets a controller reconnect without replaying history it already processed.
• Reconciliation: The act of comparing observed cluster state against registry state and issuing exactly the writes needed to make them match — an idempotent operation safe to run repeatedly.

Concepts

Now that you've seen why a static registry drifts from live cluster state, let's look at the three ideas that keep them synchronized.

The Registry Must Follow Endpoints, Not Services

A Service object is stable: its name, cluster IP, and port rarely change over its lifetime. It is tempting to sync the registry from Service objects because they map cleanly to logical service names. But a Service tells you nothing about which pods are live right now — that data lives entirely in the paired Endpoints object, which the control plane rewrites on every readiness transition. Syncing from Service alone produces a registry that knows names but routes to nothing, or worse, routes to terminated pods.

The correct source is the Endpoints object. Each carries a subsets list of ready addresses; when a pod fails its readiness probe, Kubernetes removes its IP from that list within seconds. A controller that watches Endpoints and projects each ready address into a RegistryEntry gives the mesh a routing table that tracks pod health automatically. The Service is still consulted — for its logical name and labels — but the addresses always come from Endpoints (see Code Walkthrough).

Label Selectors Make Discovery Opt-In

A cluster runs hundreds of services — system daemons, monitoring agents, one-off jobs — that have no business in the mesh routing table. Watching every Endpoints object floods the registry with noise and couples it to churn it should never see. A label selector inverts the default: nothing is discovered unless a team stamps its Service with an agreed label such as mesh-managed=true. The EndpointWatcher passes that selector to the watch call, so the API server filters server-side and streams only opted-in objects.

This opt-in boundary is also the security and blast-radius boundary. A team that has not labeled its service cannot accidentally advertise internal endpoints to the mesh, and a mislabeled object is a single-line fix rather than a registry-wide purge. The selector string is the entire contract between "runs in the cluster" and "is routable through the mesh" (see Code Walkthrough).

Reconciliation Absorbs Reconnects and Missed Events

Watch streams are not durable. Connections drop, API servers roll, and a controller can miss events during the gap. If the sync logic were a naive "apply this one delta," a missed DELETED would leave a dead address in the registry forever. Reconciliation solves this by making each event trigger an idempotent reconcile call that computes the desired set of addresses for a service and rewrites the registry entry to match — adding what's new, dropping what's gone, and doing nothing when they already agree.

Because reconcile is idempotent, the controller can recover from any reconnect by re-listing current Endpoints and reconciling each one; replaying the same object twice is harmless. This is why the watch resumes from a stored resourceVersion when possible but falls back to a full re-list without corrupting state (see Code Walkthrough).

Code Walkthrough

Having covered why the registry follows Endpoints, filters by label, and reconciles idempotently, let's implement the sync controller that ties those three ideas together. The first block defines the RegistryEntry data shape and the ServiceRegistry that stores it, exposing an idempotent reconcile method that upserts a service's addresses and a drop for deletions. The second block defines EndpointWatcher, which opens a label-filtered watch on Endpoints, extracts ready addresses from each object's subsets, and drives the registry — resuming from the last resourceVersion and re-listing on reconnect.

1from dataclasses import dataclass, field
2
3@dataclass(frozen=True)
4class Address:
5    ip: str
6    port: int
7
8@dataclass
9class RegistryEntry:
10    service_name: str
11    namespace: str
12    addresses: set[Address] = field(default_factory=set)
13
14class ServiceRegistry:
15    def __init__(self) -> None:
16        self._entries: dict[str, RegistryEntry] = {}
17
18    def _key(self, namespace: str, name: str) -> str:
19        return f"{namespace}/{name}"
20
21    def reconcile(self, namespace: str, name: str, desired: set[Address]) -> None:
22        key = self._key(namespace, name)
23        entry = self._entries.get(key) or RegistryEntry(name, namespace)
24        if entry.addresses == desired:
25            return  # already in sync; no write, no mesh churn
26        entry.addresses = desired
27        self._entries[key] = entry
28
29    def drop(self, namespace: str, name: str) -> None:
30        self._entries.pop(self._key(namespace, name), None)

The EndpointWatcher below wraps the Kubernetes Python client. Its _ready_addresses helper flattens the subsets structure into a flat set[Address], skipping not_ready_addresses. The run method opens a Watch with the label selector, dispatches each event type to reconcile or drop, and stores resourceVersion so a reconnect resumes instead of replaying.

1from kubernetes import client, watch
2
3class EndpointWatcher:
4    def __init__(self, registry: ServiceRegistry, selector: str = "mesh-managed=true") -> None:
5        self._registry = registry
6        self._selector = selector
7        self._api = client.CoreV1Api()
8        self._last_version: str | None = None
9
10    def _ready_addresses(self, ep: client.V1Endpoints) -> set[Address]:
11        result: set[Address] = set()
12        for subset in ep.subsets or []:
13            ports = [p.port for p in subset.ports or []]
14            for addr in subset.addresses or []:
15                for port in ports:
16                    result.add(Address(ip=addr.ip, port=port))
17        return result
18
19    def run(self) -> None:
20        w = watch.Watch()
21        kwargs = {"label_selector": self._selector}
22        if self._last_version:
23            kwargs["resource_version"] = self._last_version
24        for event in w.stream(self._api.list_endpoints_for_all_namespaces, **kwargs):
25            ep = event["object"]
26            ns, name = ep.metadata.namespace, ep.metadata.name
27            self._last_version = ep.metadata.resource_version
28            if event["type"] == "DELETED":
29                self._registry.drop(ns, name)
30            else:
31                self._registry.reconcile(ns, name, self._ready_addresses(ep))

Verify by labeling one Service with mesh-managed=true, scaling its deployment from 2 to 3 replicas, and confirming the matching RegistryEntry.addresses set grows to three Address entries within seconds — then deleting a pod and watching the set shrink back.

Do's and Don'ts

Having walked through the sync controller above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do source addresses from Endpoints subsets, not from Service objects — the _ready_addresses helper reads live, health-filtered pod IPs, whereas a Service only names the target; syncing from the Service alone yields a registry that routes to pods that may already be terminated.
2. ✓Do filter the watch with a label selector so discovery is opt-in — passing mesh-managed=true to EndpointWatcher means only deliberately labeled services enter the registry, keeping system daemons and one-off jobs out of mesh routing tables and bounding the blast radius of a mislabel to a single object.
3. ✓Do make reconcile idempotent by comparing the desired address set to the current one — returning early when the sets are equal suppresses redundant registry writes and the mesh sidecar reloads they trigger, and it makes replaying a watch event after reconnect completely harmless.

Don'ts

1. ✗Don't poll list_endpoints_for_all_namespaces on a timer instead of using a watch stream — full-list polling scales with cluster size and adds seconds of staleness on every interval, while the run loop's watch pushes ADDED/MODIFIED/DELETED deltas the instant kube-controller-manager rewrites an Endpoints object.
2. ✗Don't ignore the resourceVersion token when reconnecting — dropping it forces the API server to replay the entire object set on every reconnect; storing _last_version and passing it back resumes the stream past what the watcher already reconciled, avoiding a redundant re-sync storm.
3. ✗Don't include not_ready_addresses when building the Address set — pods that are starting up or failing readiness appear in that list, and projecting them into a RegistryEntry advertises endpoints the mesh will route to before they can serve traffic, producing connection-refused errors the moment a new replica is scheduled.