Network Isolation

Default deny, in every tenant namespace

Kubernetes doesn’t have a cluster-wide “default deny” for network traffic, so each namespace needs its own policies. A good baseline for each tenant namespace is:

• Default deny for both ingress and egress.
• Allow traffic within the namespace if the tenant’s applications need it.
• Allow egress to cluster DNS.
• An explicit block on the cloud metadata service (169.254.169.254), which is reachable by default and often hands out node-level cloud credentials.

It’s much easier to put these in place before any workloads are deployed than to retrofit them later. When you’re retrofitting, observability data from something like OpenTelemetry is really useful for working out what traffic actually needs allowing.

Keep it out of tenants’ reach

There’s a catch with namespaced NetworkPolicies in a multi-tenant cluster. The default admin and edit roles can write NetworkPolicy objects, so a tenant can delete or loosen the baseline policies in their own namespace.

If your CNI supports them, cluster-wide policy types like Cilium’s CiliumClusterwideNetworkPolicy or Calico’s GlobalNetworkPolicy are a better home for the tenant baseline, as tenants can’t edit them. Tenants can then add their own namespaced policies on top.

A couple of other things to check:

• The CNI actually enforces policy. Some don’t (Flannel, kindnet), and a policy that isn’t enforced fails silently. Test with real traffic, not just kubectl get netpol.
• Namespace selectors use identity. Select other namespaces with the kubernetes.io/metadata.name label, not a custom label a tenant could add to their own namespace.
• hostNetwork is blocked by admission control, as host networked pods sit outside policy enforcement.
• Operators have egress policies too. Most operators only need to talk to the API server and a couple of known endpoints.

DNS

Network policy doesn’t stop DNS enumeration, as tenants only need to talk to the cluster DNS server. There are a couple of things that can help:

• Switch CoreDNS from pods insecure to pods verified. The default pods insecure setting will return an A record for any IP address embedded in a *.pod.cluster.local name, without checking a pod exists. Fabian Kammel showed that this can be used to bypass FQDN-based egress policies that trust the cluster domain.
• The CoreDNS policy plugin can restrict lookups by namespace, although I’ve not seen it used in practice.
• Virtual clusters give each tenant their own DNS.

Shared network components

The shared paths that network policy can’t see need fixing separately:

• Block ExternalIP services with DenyServiceExternalIPs (CVE-2020-8554).
• Remove Endpoints/EndpointSlices write from tenants (CVE-2021-25740). New clusters don’t grant it in the default roles, but upgraded clusters may still have it.
• Per-tenant ingress controllers limit what a tenant can do by tricking a shared one, and avoid cross-namespace annotation tricks like the Traefik one.
• Keep service mesh config with the platform team. Don’t give tenants networking.istio.io rights. If they need to configure routing, the Gateway API was designed with cross-namespace references in mind.

What network isolation doesn’t fix

• The API server proxy. Proxied requests come from the control plane, not the tenant’s pod, so SSRF through the API server can still reach other tenants’ workloads unless the control plane’s network access is restricted too.
• Node compromise. Anyone who gets onto a node can see cluster-wide service and endpoint information through the node’s credentials, and may be able to attack the network from there.
• Service discovery, unless you’ve also changed how DNS works.