Service discovery and configuration

A service that calls another needs two things it should never hard-code: where the other service is, and what its own settings are in this environment. Discovery answers the first and has mostly become a platform feature; configuration answers the second and is where secrets, rotation and feature flags live. This lesson is the discovery patterns and which one you are actually using on Kubernetes, configuration as a source with an order of precedence, secrets that rotate without a redeploy, and feature flags as the thing they are.

#Discovery patterns

Instances come and go: they scale, restart, move hosts. Something has to map "the inventory service" to a set of addresses that is correct right now.

Client-side discovery. Each instance registers itself with a registry (Eureka, Consul) and sends heartbeats; a caller asks the registry for the instances and load-balances among them itself. Spring Cloud's @LoadBalanced RestClient.Builder with a Eureka client does this: http://inventory/stock/42 resolves through the registry. The caller holds the logic, which means every language and framework needs a client, and the registry is a piece of infrastructure to run and to keep consistent.

Server-side discovery. The caller sends to a stable name; a load balancer or the platform resolves it to a healthy instance. The caller knows nothing about registries. This is what cloud load balancers do, and it is what Kubernetes does for every service by default, which is why client-side registries have largely disappeared from new systems.

Service mesh. A sidecar proxy (Istio, Linkerd) or a node-level proxy (Istio's ambient mode) intercepts every call, does discovery, load balancing, retries, mutual TLS and metrics uniformly, and the application makes plain HTTP calls to plain names. It moves resilience out of every codebase into one layer, at the cost of a control plane and per-hop latency. Right at a certain scale of services and teams; overkill for six services and one team.

#DNS in Kubernetes

A Kubernetes Service is a stable name and a virtual IP in front of a set of pods selected by label. Inside the cluster, inventory resolves within the same namespace, and inventory.shop.svc.cluster.local from anywhere; the platform's DNS returns the Service's IP, and kube-proxy (or the CNI) routes to a ready pod. A pod is "ready" when its readiness probe passes, which is the Actuator readiness group from the Spring Boot course; a pod that is not ready receives nothing. This is server-side discovery with no registry to run and no client library.

spring:
  application.name: orders
inventory:
  base-url: http://inventory.shop.svc.cluster.local:8080    # a Service; the platform picks the pod

Two details that matter. Service IPs load-balance per connection, not per request, so a caller with a keep-alive pool to one pod keeps sending to that pod; an HTTP client that closes idle connections after a short time, or a mesh, spreads the load. And a headless Service (clusterIP: None) returns the pod IPs themselves, which is what a client that must talk to a specific instance (Kafka clients, a database's replicas) needs.

#Configuration

Spring Boot's precedence order (the Configuration properties lesson) already gives one model: defaults in the jar, overridden by the environment. On Kubernetes the environment is a ConfigMap mounted as environment variables or as files, and per-environment values live there, versioned alongside the deployment manifests, not in the image. That covers most services.

Spring Cloud Config Server centralises configuration in a git repository and serves it to services at startup over HTTP, with spring.config.import=configserver:; it predates ConfigMaps and remains useful when services run outside Kubernetes, when configuration must be shared across many services and changed in one place, or when audit history in git is a requirement. It is one more service to run and to keep available at every other service's startup; with optional:configserver: a missing server does not stop the boot.

Changing configuration at runtime. A ConfigMap change does not reach a running process by itself: environment variables are read at start, and mounted files update on the pod but nothing rereads them. Options in order of preference: a rolling restart (configuration changes are deploys, and a restart is the honest way to apply one); Spring Cloud's @RefreshScope with a /actuator/refresh call or a Spring Cloud Bus event, which re-creates the annotated beans with new values; or Spring Cloud Kubernetes watching ConfigMaps. A setting that must change without a restart is usually a feature flag in disguise.

#Secrets

A secret is configuration whose value must not be seen, and the difference in handling is total:

• Not in the image, not in git, not in a ConfigMap. A Kubernetes Secret object is base64 (encoding, not encryption); it is protected by RBAC and, if enabled, encryption at rest, and it is mounted as a file or an environment variable like any other config. Prefer files: environment variables leak into child processes, crash dumps and /proc.
• A vault as the source of truth. HashiCorp Vault, AWS Secrets Manager or the cloud equivalent holds the secret; the External Secrets Operator or a CSI driver syncs it into the cluster's Secret objects, so the cluster never holds the master copy and rotation happens at the source. Spring Cloud Vault reads directly from Vault at startup when the cluster layer is not wanted.
• Rotation. A database password rotated in the vault is useless until the service reads the new one. Three approaches: short-lived dynamic credentials (Vault issues a database user that expires in an hour, and the client renews), a rolling restart triggered by the rotation, or a connection pool that reloads its credentials on the next connection (HikariCP with a DataSource that re-reads the mounted file, or a driver plugin). Choose one before the first rotation, because the first rotation is otherwise an outage.
• Never log the bound properties object, never expose /actuator/env unmasked, and never let a secret into a URL where it lands in access logs.

#Feature flags

A feature flag is configuration that changes behaviour rather than environment: is the new checkout on, for whom, and can it be turned off in ten seconds without a deploy. That is a different thing from a database URL and deserves a different tool: Togglz or OpenFeature with a provider (LaunchDarkly, Unleash, Flagsmith), evaluated per request with a context (user, tenant, percentage), changed from a UI, and audited.

Rules that keep flags from becoming the codebase's second configuration system: a flag has an owner and an expiry, and it is removed when the feature is fully on; a kill switch for a risky dependency is a flag worth keeping permanently; flags are evaluated at the edge of a use case, not scattered through it; and the number of live flags is a metric someone watches, because fifty long-lived flags is fifty untested combinations.

#Under the hood: what a Service name resolves to, and how a secret reaches the pod

A Kubernetes Service is an object with a selector and a stable ClusterIP; the endpoint controller watches pods matching the selector and writes an EndpointSlice listing the IPs of the ones whose readiness probe passes. CoreDNS answers inventory.shop.svc.cluster.local with the ClusterIP (not the pods), and kube-proxy on every node programs iptables or IPVS rules that rewrite a packet to the ClusterIP into a packet to one of the endpoint IPs, chosen at random per new connection. The rewrite is a NAT rule, which is why load balancing is per connection: a client holding a keep-alive connection sends every request through the same rule to the same pod, and a pool of ten idle connections may all point at one replica. Readiness is the whole control: a pod failing its probe is removed from the slice within a probe period, kube-proxy drops the rule, and new connections stop; existing connections are not cut, which is what the preStop sleep in the Spring Boot course protects. A headless Service skips the ClusterIP and has CoreDNS return the pod IPs directly, so the client sees the members, which is what a Kafka client or a stateful set's peers need. A mesh replaces the NAT rule with a proxy (Envoy) that sees every request, so balancing becomes per request, retries and timeouts move into the proxy's routing rules, and mTLS is terminated there with certificates issued by the mesh's own CA and rotated hourly.

Configuration reaches a process by two routes and they behave differently. Environment variables are read from a ConfigMap or Secret at container start, copied into the process's environment, and never updated; they are also inherited by every child process and visible in /proc/<pid>/environ and in crash dumps. A volume mount from a ConfigMap or Secret is a directory the kubelet keeps in sync: it writes the new version to a timestamped directory and swaps a symlink, atomically, within the kubelet's sync period (about a minute), so a process that re-reads the file sees the update, and spring.config.import=configtree: binds such a directory at startup. subPath mounts do not update, which is a common surprise. A Secret is stored in etcd base64-encoded, encrypted at rest only if the API server is configured with an encryption provider, and readable by any principal with get secrets in the namespace; the External Secrets Operator or the Secrets Store CSI driver fetches from a vault with a workload identity and writes or mounts it, so the vault, not etcd, is the system of record, and rotation is a sync.

Spring Cloud Config's @RefreshScope is a bean scope: the annotated bean is a proxy over a target that lives in a scope cache, and /actuator/refresh (or a RefreshRemoteApplicationEvent from the bus) clears the cache and re-binds @ConfigurationProperties, so the next call to the proxy constructs a new target with the new values. Anything that captured the old target's state (a DataSource already built from the old password) is not refreshed unless it too is in the scope, which is why credential rotation through refresh works only if the pool itself is rebuilt, and why most teams restart instead.

#Walkthrough: the rotation that took the service down at 3 am

A database password was rotated on schedule in Vault. The rotation itself was flawless.

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef: { name: orders-db, key: password }
---
# ExternalSecret syncing vault:secret/orders/db → Secret orders-db, refreshInterval: 1h
# HikariCP: maximum-pool-size 10, max-lifetime 30m

1. Vault rotated the password and, with the database plugin's rotation, set the new one on the database user. The ExternalSecret synced within the hour; the Secret object updated. The running pods had the old password in their environment, copied at start, and their pooled connections, already authenticated, kept working.
2. Thirty minutes later, max-lifetime retired the first pooled connection and Hikari opened a replacement with the old password from the environment. FATAL: password authentication failed. Over the next thirty minutes every connection was retired and every replacement failed; the pool shrank to zero; the service was down at 3 am with no deploy and no alert about secrets, and the on-call engineer's first hypothesis was the database.
3. Nothing in the chain was wrong: the vault rotated, the operator synced, Kubernetes updated the Secret. The environment variable was the break: a snapshot taken at start that no component was responsible for refreshing.
4. The fix, chosen from three: the Secret mounted as a file (/etc/secrets/db-password), and a DataSource whose password supplier reads that file on each new connection (Hikari's setPassword is honoured for new connections; a small DataSource wrapper, or the driver's credential plugin, does it). The alternatives were a rolling restart triggered by the operator's reloader annotation on rotation, or dynamic database credentials with a lease that Spring Cloud Vault renews and re-binds. The team took the file-and-supplier route because it needs no restart and no new dependency, and set max-lifetime well below the rotation period so every connection would be re-authenticated with the new value long before the old one was revoked.
5. Staging now rotates the secret nightly, and a test asserts the pool survives the rotation; the alert is on hikaricp.connections.timeout rate, which fires minutes before the pool empties.

A rotated secret is only rotated when every reader has the new value; an environment variable is a reader that never reads again.

#Try it yourself

#Misconceptions

• "Kubernetes load-balances requests across pods." kube-proxy balances connections; a keep-alive client sticks to a pod. Per-request balancing needs a mesh or client-side balancing.
• "A Secret is encrypted." It is base64 in etcd, encrypted at rest only if configured, readable by anyone with get secrets. The vault is the safe; the Secret is a delivery mechanism.
• "Updating a ConfigMap updates the app." Environment variables never update; mounted files update but are not re-read; subPath mounts do neither. Restart, or refresh deliberately.
• "Rotating a secret in the vault rotates it." It is rotated when every reader has the new value and every cached credential has been re-issued. Design that path before the first rotation.
• "A service registry is still needed." On Kubernetes the Service, EndpointSlice and DNS are the registry; Eureka-style clients add a second, stale view of the same thing.

#Going deeper

• Kubernetes documentation: "Service", "EndpointSlices", "DNS for Services and Pods", "Configure a Pod to Use a ConfigMap" (the section on mounted files updating), and "Secrets" (the security properties section).
• External Secrets Operator and Secrets Store CSI Driver documentation; HashiCorp Vault's "Database secrets engine" for dynamic credentials.
• Spring Cloud Commons, "Refresh Scope", and Spring Cloud Kubernetes "ConfigMap PropertySource" reload modes.
• Istio, "Traffic Management" and "Security → Mutual TLS", for what the sidecar takes over.
• OpenFeature specification and the Togglz documentation; Pete Hodgson, "Feature Toggles (aka Feature Flags)" on martinfowler.com.