Separate Clusters

Adding it all up

If you’ve read through the rest of the hardening section, you’ll have noticed a pattern. Each control fixes part of the problem and leaves something else shared. So a cluster for tenants who really don’t trust each other ends up needing:

• Admission control to stop privileged pods,
• careful RBAC and operator hygiene,
• network isolation with tenant-proof baseline policies,
• node pools to separate the nodes,
• sandboxed runtimes in case of kernel exploits,
• and virtual clusters to separate the control plane.

At that point, it’s fair to ask whether that’s really cheaper or easier than just running separate clusters. For a lot of setups, I don’t think it is.

This isn’t a new conclusion either. When Atredis assessed Salesforce’s “shared control plane, isolated worker nodes” design back in 2021, they didn’t find any critical issues, but still recommended looking at Clusters as a Service, as the blast radius of a control plane compromise could be unacceptable for high-security environments.

Making it manageable

The main argument against separate clusters is the management overhead, which was one of the reasons for multi-tenancy in the first place. That’s a lot more manageable than it used to be, though:

• Tools like Cluster API and Rancher are designed for creating and managing large numbers of clusters.
• Managed Kubernetes takes the control plane off your hands, although your tenants are then relying on the provider’s own multi-tenancy, as provider control planes are a shared environment too.

It doesn’t have to be all or nothing

The multi-tenancy spectrum is useful here. A couple of applications from the same team can quite reasonably share a cluster, and different internal teams can probably share one with some effort. External customers who might be actively hostile are the case where a cluster each makes the most sense.

So a practical approach is to draw the cluster boundary at the trust boundary: share where tenants trust each other, and split where they don’t.

What separate clusters don’t fix

Sharing doesn’t stop at the edge of the cluster. Separate clusters can still share a container registry, a CI/CD system, a cloud account and its IAM, or a network. Each of those is a shared surface in its own right, and needs the same kind of thought as the shared surfaces inside a cluster.

Conclusion

Multi-tenancy in Kubernetes is tricky to do well. The platform was designed to share infrastructure between workloads, not to keep tenants who don’t trust each other apart, and almost everything tenants share is hard to isolate. For soft multi-tenancy, the techniques on this site will get you a long way. For hard multi-tenancy, separate clusters are often the better plan.