Services and NetworkingServices and Networking

Part of the Kubernetes Roadmap knowledge base.

Overview

Networking is where Kubernetes stops being “Docker with a scheduler” and becomes a distributed system. Pods are ephemeral — they come and go, get rescheduled onto other nodes, and change IPs every time. Yet applications need stable addresses, load balancing, service discovery, and controlled exposure to the outside world. Kubernetes networking exists to bridge that gap: it gives every Pod a real IP, lets any Pod talk to any other Pod without NAT, and layers Services, DNS, Ingress, and NetworkPolicy on top to provide stable naming, traffic distribution, external access, and security segmentation.

The design is deliberately simple at the base and pluggable above it. The base is the Kubernetes network model — a small set of requirements every cluster must satisfy. Everything else (how packets actually move, which load-balancer gets provisioned, how DNS resolves) is implemented by pluggable components: a CNI plugin for Pod networking, kube-proxy (or an eBPF dataplane) for Service routing, CoreDNS for name resolution, and an Ingress/Gateway controller for HTTP routing. Understanding which layer owns which responsibility is the key to debugging networking, because a “connection refused” can originate in any one of them.

A mental model

Think of Kubernetes networking as four decoupled layers stacked on the flat Pod network. Layer 0 — Pod network: every Pod has an IP and can reach every other Pod directly (the CNI’s job). Layer 1 — Service: a stable virtual IP + DNS name that load-balances to a changing set of Pods (kube-proxy’s job). Layer 2 — DNS: turns names like web.prod.svc.cluster.local into Service IPs (CoreDNS’s job). Layer 3 — Ingress/Gateway: L7 HTTP routing and TLS from outside the cluster (the controller’s job). Layer 4 — NetworkPolicy: a firewall that decides which of those flows are actually allowed (the CNI’s job again). Each layer depends on the one below but is otherwise independent — and each can fail independently.

Fundamentals

The Kubernetes network model

Kubernetes does not implement networking itself; it mandates a model and lets CNI plugins satisfy it. Every conforming cluster guarantees:

1. Every Pod gets its own unique, cluster-wide IP address. A Pod’s containers share that IP (and a network namespace), reaching each other over localhost.
2. Every Pod can communicate with every other Pod on any node without NAT. The network is flat — Pod IPs are routable across the whole cluster.
3. Agents on a node (kubelet, system daemons) can reach all Pods on that node.
4. The IP a Pod sees itself as is the same IP others use to reach it. No source-NAT rewriting inside the Pod network.

This “IP-per-Pod, flat network, no NAT” model is what makes Kubernetes networking feel uniform. It removes the port-mapping gymnastics of plain Docker (-p 8080:80) — inside the cluster, a Pod just listens on its real port and is reachable at its real IP. The trade-off is that it demands real routing infrastructure, which is precisely what the CNI plugin provides.

The problem Services solve

Pod IPs are unstable: scale up, roll out a new version, or lose a node, and the set of IPs backing your app changes completely. You cannot hardcode Pod IPs. A Service is a stable abstraction in front of a dynamic set of Pods — a fixed virtual IP (the ClusterIP) and DNS name that never changes for the life of the Service, load-balancing to whichever Pods currently match its label selector. The Service continuously tracks its backends through EndpointSlices, so as Pods appear and disappear the Service’s target list updates automatically.

How kube-proxy implements Services

A ClusterIP is virtual — no process listens on it, and no single machine owns it. kube-proxy, a DaemonSet-like agent on every node, watches Services and EndpointSlices and programs the node’s kernel so that packets destined for a Service IP are transparently rewritten (DNAT) to a real Pod IP. There are three modes:

The important consequence: Service load balancing happens in the kernel of the client’s node, not at a central proxy. There is no bottleneck box; each node independently forwards to a backend. It is also connection-level L4 load balancing (per TCP connection), not L7 — long-lived connections (gRPC, HTTP/2) pin to one backend and can unbalance, which is a common gotcha solved by client-side balancing or a service mesh.

EndpointSlices

Behind every selector-based Service, Kubernetes maintains the list of ready backend IPs. Originally this was a single Endpoints object per Service; a 5000-Pod Service meant one enormous object rewritten on every Pod change, overwhelming etcd and every watcher. EndpointSlices (stable since v1.21) shard that list into chunks of ~100 endpoints, so a single Pod change rewrites only one small slice. They also carry richer per-endpoint data: topology hints (for topology-aware routing that prefers same-zone backends), readiness, and per-port info. EndpointSlices are the source of truth kube-proxy actually reads.

Key Concepts

Service types

NodePort and LoadBalancer are supersets: a LoadBalancer Service also allocates a NodePort, and the cloud LB forwards to that NodePort on the nodes. So the layering is LoadBalancer → NodePort → ClusterIP → Pods.

Headless Services and stable Pod identity

A normal Service hides its Pods behind one virtual IP. A headless Service (clusterIP: None) does the opposite: a DNS query for the Service returns the A/AAAA records of all ready Pods directly, and — when combined with a StatefulSet — each Pod also gets its own stable DNS name: pod-0.web.prod.svc.cluster.local, pod-1.web..., etc. This is essential for stateful systems (databases, Kafka, etcd) where clients must address specific replicas (e.g., the primary), not a random one. The client does its own load balancing or peer discovery from the returned IPs.

DNS in the cluster (CoreDNS)

Every cluster runs a DNS server — CoreDNS (a flexible, plugin-based server; the modern default, replacing kube-dns). The kubelet configures every Pod’s /etc/resolv.conf to point at CoreDNS’s ClusterIP and injects search domains, so short names resolve. The naming scheme:

• Service A record: <service>.<namespace>.svc.<cluster-domain> → the Service’s ClusterIP. Default cluster domain is cluster.local.
• Short names: from a Pod in namespace prod, web resolves to web.prod.svc.cluster.local (same namespace), and web.staging to web.staging.svc.cluster.local, thanks to the search domains in resolv.conf.
• Headless Service: returns multiple A records (one per ready Pod) instead of a single ClusterIP.
• Pod DNS (StatefulSet): <pod-name>.<service>.<namespace>.svc.cluster.local gives each replica a stable name.
• SRV records: _<port-name>._<proto>.<service>.<namespace>.svc.cluster.local for named ports.

# From inside a Pod
nslookup web                       # short name, same namespace
nslookup web.prod.svc.cluster.local
dig +short db-0.db.prod.svc.cluster.local   # a specific StatefulSet replica

DNS is the backbone of service discovery: applications connect to names, not IPs, and Kubernetes keeps the name→IP mapping current. When “the app can’t reach the database,” DNS resolution inside the Pod is one of the first things to check.

Ingress and Ingress controllers

A LoadBalancer Service gives you one external IP per Service — expensive and L4-only. Ingress solves L7 HTTP(S) routing: a single entry point that routes by host and path to many backend Services, terminates TLS, and consolidates external exposure behind one load balancer.

Critically, the Ingress resource is just configuration — it does nothing on its own. You must run an Ingress controller (a Pod that watches Ingress objects and programs an actual proxy). Common controllers: ingress-nginx (NGINX-based, the most widely deployed), Traefik, HAProxy, Contour (Envoy-based), and cloud-native ones (AWS Load Balancer Controller, GKE Ingress). You select which controller handles an Ingress via ingressClassName.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod   # auto-provision TLS
spec:
  ingressClassName: nginx
  tls:
    - hosts: [shop.example.com]
      secretName: shop-tls          # cert stored here (by cert-manager)
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service: { name: api, port: { number: 80 } }
          - path: /
            pathType: Prefix
            backend:
              service: { name: web, port: { number: 80 } }

pathType matters: Prefix matches path segments (/api matches /api/v1), Exact matches the whole path literally, ImplementationSpecific defers to the controller. Advanced behavior (rewrites, rate limits, auth, canary) is configured through controller-specific annotations — which is exactly the fragmentation the Gateway API was created to fix.

Gateway API — the successor to Ingress

Ingress hit a ceiling: it only really models HTTP host/path routing, and everything beyond that (TLS passthrough, header matching, traffic splitting, gRPC) leaked into non-portable annotations. The Gateway API (GA since v1.0, an official Kubernetes SIG-Network project) is the modern replacement. It is role-oriented — it splits responsibilities across resources owned by different teams:

This separation lets platform teams own the Gateway (and its TLS/security) while app teams manage their own routes without touching shared infrastructure. It supports traffic splitting/weighting, header/method matching, and cross-namespace routing natively — no annotation soup. It is protocol-extensible and is where the ecosystem is heading; new implementations (Envoy Gateway, Cilium, Istio, NGINX Gateway Fabric) target it. Treat Ingress as stable-but-frozen and Gateway API as the forward path.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: shop }
spec:
  parentRefs: [{ name: prod-gateway }]
  hostnames: ["shop.example.com"]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /api } }]
      backendRefs: [{ name: api, port: 80 }]
    - matches: [{ path: { type: PathPrefix, value: / } }]
      backendRefs:                      # canary: 90/10 traffic split
        - { name: web, port: 80, weight: 90 }
        - { name: web-next, port: 80, weight: 10 }

NetworkPolicy

By default, the Kubernetes network is flat and fully open: any Pod can connect to any other Pod, in any namespace. That is a security liability — a compromised frontend can reach your database directly. A NetworkPolicy is a namespaced firewall that selects Pods by label and defines allowed ingress (inbound) and egress (outbound) traffic.

The rules are additive and the logic is important:

• A Pod is “unselected” (fully open) until some NetworkPolicy selects it. Once any policy selects a Pod for a direction (ingress/egress), that direction flips to default-deny, and only explicitly allowed traffic passes.
• Policies are allow-only — there is no “deny” rule; you deny by not allowing.
• NetworkPolicy is enforced by the CNI plugin. A cluster whose CNI doesn’t support it (e.g., plain Flannel) silently ignores the policies — a dangerous false sense of security.

A common baseline is a default-deny-all policy per namespace, then explicit allows:

# 1. Default-deny all ingress AND egress in namespace "prod"
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: prod }
spec:
  podSelector: {}                 # selects every Pod in the namespace
  policyTypes: [Ingress, Egress]  # both directions now default-deny
---
# 2. Allow the api Pods to receive traffic only from web Pods, on 8080
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-allow-web, namespace: prod }
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: web } }
      ports:
        - { protocol: TCP, port: 8080 }
---
# 3. Allow all Pods to resolve DNS (egress to CoreDNS) — almost always needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: prod }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

The classic pitfall: apply a default-deny-egress policy and suddenly every Pod’s DNS breaks, because CoreDNS is now blocked. Always pair egress lockdown with an explicit DNS allow (rule 3). For richer policies (egress to FQDNs, L7 rules, cluster-wide policy), CNIs like Cilium and Calico offer their own CRDs (CiliumNetworkPolicy, Calico GlobalNetworkPolicy) beyond the core spec.

CNI plugins

The Container Network Interface (CNI) is the standard the kubelet uses to set up Pod networking. When a Pod starts, the kubelet calls the CNI plugin, which assigns the Pod an IP and wires its network namespace into the cluster network. The CNI you choose determines performance, the encapsulation used, and whether NetworkPolicy is enforced.

Overlays (VXLAN) encapsulate Pod traffic in UDP so it can traverse any underlying network at some CPU/MTU cost; native routing (Calico BGP, AWS VPC CNI) avoids encapsulation for lower overhead but requires the underlying network to route Pod IPs. For most new clusters the choice is Cilium (eBPF, observability, replaces kube-proxy) or Calico (proven, flexible); Flannel remains popular for its simplicity when you don’t need NetworkPolicy.

Service discovery, end to end

Putting the layers together, here is how a request finds its target:

1. App connects to a name (http://api) — never a hardcoded IP.
2. CoreDNS resolves api → api.prod.svc.cluster.local → the Service’s stable ClusterIP.
3. The client node’s kube-proxy (iptables/IPVS/eBPF) DNATs the ClusterIP to a real Pod IP, chosen from the Service’s current EndpointSlices.
4. The CNI routes the packet across the flat Pod network to the target node/Pod.
5. NetworkPolicy (enforced by the CNI) allows or drops the flow.

If any step is misconfigured the connection fails, and each step has a distinct symptom — which is why methodical, layer-by-layer debugging beats guessing.

Best Practices

1. Connect to Service DNS names, never Pod IPs. Pod IPs are ephemeral; Service names are stable. Hardcoding IPs breaks on the first reschedule and defeats the entire abstraction.
2. Prefer ClusterIP + Ingress/Gateway over per-Service LoadBalancers. One LoadBalancer per Service is costly and L4-only; consolidate HTTP exposure behind a single Ingress/Gateway with host/path routing and shared TLS.
3. Adopt a default-deny NetworkPolicy baseline, then allow explicitly. The network is open by default; lock each namespace down to least-privilege and open only required flows — this contains lateral movement after a compromise.
4. Always allow DNS egress when locking down egress. A default-deny-egress policy silently breaks CoreDNS resolution; pair it with an explicit UDP/TCP 53 allow to kube-system, or nothing resolves.
5. Verify your CNI actually enforces NetworkPolicy. Plain Flannel ignores policies silently — writing them gives false security. Use Calico/Cilium (or Flannel+Calico) where policy matters.
6. Set readiness probes so Services only route to ready Pods. Endpoints reflect readiness; a Pod without a readiness probe receives traffic before it can serve it, causing errors during rollouts and scale-ups.
7. Choose the right Service type deliberately. ClusterIP for internal, Ingress/Gateway for HTTP ingress, LoadBalancer only for non-HTTP or raw L4, Headless for StatefulSets/client-side LB. NodePort is mostly a dev/behind-LB primitive, not a production front door.
8. Migrate new external routing to the Gateway API. Ingress is stable but frozen and annotation-bound; Gateway API gives portable, role-separated, traffic-splitting-capable routing. Start greenfield services there.
9. Use Headless Services for stateful workloads. Databases and clustered systems need per-Pod stable DNS identity (pod-0.svc…), which a normal ClusterIP hides. Pair StatefulSets with a headless Service.
10. Automate TLS with cert-manager. Don’t hand-manage certificates on Ingress/Gateway; cert-manager provisions and renews (e.g., Let’s Encrypt) automatically, eliminating expiry outages.
11. Prefer IPVS or eBPF for large clusters. iptables Service rules scale O(n); at thousands of Services, switch kube-proxy to IPVS or adopt Cilium’s eBPF dataplane to keep rule programming fast.
12. Beware L4 load balancing for HTTP/2 and gRPC. kube-proxy balances per-connection; long-lived multiplexed connections pin to one backend. Use client-side load balancing, headless Services, or a service mesh to spread that traffic.
13. Label Pods consistently and meaningfully. Services, NetworkPolicies, and EndpointSlices all key off labels; an inconsistent or missing label silently removes a Pod from a Service or a policy’s scope.
14. Restrict LoadBalancer/Ingress source ranges. Use loadBalancerSourceRanges or WAF/ingress annotations to limit who can reach public endpoints; don’t expose admin or internal APIs to 0.0.0.0/0.
15. Test policies and routing before rollout. Validate NetworkPolicies with connectivity tests (Cilium connectivity test, kubectl exec curl) and Ingress rules in staging; a wrong selector can either black-hole traffic or leave it wide open.
16. Give CoreDNS resources and autoscaling. DNS is on the critical path of every request; under-provisioned CoreDNS causes cluster-wide latency and intermittent resolution failures. Monitor it and scale it with cluster size.

References

• Kubernetes — Cluster Networking (the network model)
• Kubernetes — Service
• Kubernetes — EndpointSlices
• Kubernetes — DNS for Services and Pods
• Kubernetes — Ingress · Ingress Controllers
• Kubernetes — Network Policies
• Kubernetes — Virtual IPs and Service Proxies (kube-proxy modes)
• Gateway API
• ingress-nginx · Traefik · Contour
• Calico · Cilium · Flannel
• CoreDNS · cert-manager
• CNCF landscape — networking
• roadmap.sh — Kubernetes