Virtual Clusters

How they work

With virtual clusters, each tenant gets their own control plane (an API server, controller manager and datastore) running as workloads in a namespace on a shared host cluster. The tenant talks to their own API server, and when they create pods, a syncer copies them down to the host cluster, which actually schedules and runs them on its nodes.

vCluster is the best-known open source implementation.

What virtual clusters fix

Most of the API server problems come from tenants sharing one control plane, so giving each tenant their own fixes a lot of them:

• Cluster-scoped resources. Each tenant has their own CRDs, webhook configurations and so on, so two tenants can run different versions of the same operator.
• DNS. Each virtual cluster has its own DNS, so tenants can’t enumerate each other’s services.
• Information leakage. Listing namespaces in your virtual cluster only shows you your own.
• The RBAC juggling act. Tenants can be cluster-admin inside their own virtual cluster, so you don’t need to carve out a careful set of namespace rights for them.

What virtual clusters don’t fix

• Nodes. Tenant pods still run on the host cluster’s nodes, sharing the kernel with everyone else’s pods, so container breakouts are still a concern. For tenants who don’t trust each other, virtual clusters need to be combined with node pools and/or sandboxed runtimes.
• The host network. Pods are running on the host cluster’s network, so network isolation still needs to be done at the host level.
• The syncer. The component that copies objects between the virtual and host clusters is itself a privileged, shared piece of software acting on tenant input, which is exactly the shape of thing that turns up on the operators page. How well virtual clusters actually isolate tenants in practice is something I’d like to look at in more detail.

It’s also worth checking exactly what any given implementation installs and connects to. In late 2024 the vCluster install process hooked into the Loft online platform by default, which might not be what you want for your cluster management plane.

Is it worth it?

Virtual clusters sit nicely in the middle of the tenancy models. They fix a lot of the control plane problems for a lot less cost than separate clusters. But by the time you’ve added node pools, sandboxed runtimes and host-level network isolation to cover what they don’t fix, it’s worth asking whether separate clusters would be simpler.