Hardening · Workload isolation
Node Pools
Dedicated node pools keep selected tenants off the same worker nodes. They replace one cluster-wide host trust domain with smaller, more deliberate ones.
A node pool is a group of worker nodes configured and managed together. Giving a tenant or trust class its own pool means its workloads no longer share a kernel with tenant workloads in other pools. A container escape on one of those nodes has a smaller immediate blast radius.
Node pools are especially useful when tenants have different risk profiles, or when a sensitive workload cannot accept the shared kernel boundary described in the Nodes risk. They also help with noisy neighbours, as tenants on separate nodes aren’t fighting over the same CPU and memory. This does have a knock-on reduction in the cost benefits of Kubernetes though, as you lose some of the ability to pack workloads tightly onto shared nodes.
Making it stick
Node pools are usually implemented with taints on the nodes, plus tolerations
and a nodeSelector or node affinity on the tenant’s pods. The catch is that
those fields are all part of the pod spec, so a tenant who can write their own
manifests can just add a toleration and schedule onto another tenant’s nodes.
To make the separation hold, placement needs to be enforced rather than
requested, using admission control. Something like the
PodTolerationRestriction
admission controller, or a policy engine, can make sure each tenant’s pods only
ever land on that tenant’s nodes.
What node pools don’t fix
Node pools are a good control, but they don’t remove every shared surface, so it’s worth being clear about what’s left.
- DaemonSets still run everywhere. Cluster-wide DaemonSets put a pod on every node in every pool, so their service account tokens are available to anyone who escapes onto any node. See the operators page.
- Nodes can still see the rest of the cluster. The Atredis assessment of Salesforce’s shared control plane model found that a compromised node could still list pods, services and nodes across every namespace, even with the Node Authorizer and NodeRestriction in place.
- The API server and network are still shared. Node pools don’t do anything about the API server or networking risks.
For tenants that really don’t trust each other, node pools work best combined with sandboxed runtimes (gVisor or Kata Containers) and careful screening of what DaemonSets and operators are running in the cluster.