Sandboxed Runtimes

Why this matters more now

Containers are just processes. A standard container runtime like runc uses namespaces, capabilities, seccomp and LSMs to fence a process off, but it’s still running on the same Linux kernel as everything else on the node. A kernel bug can take you straight through all of those layers.

That’s always been true, but as I covered on the Nodes page, the cost of turning a kernel LPE into a working container breakout has dropped a lot now that LLMs can do most of the work. If a tenant can run code in a container and you don’t trust them, it’s safest to assume they can reach the node, and design with that in mind.

The options

There are two main alternatives to standard containers:

• Sandboxed runtimes like gVisor, which intercept the container’s system calls and handle them with a user-space kernel written in Go, so the workload talks to the host kernel a lot less directly.
• VM-based runtimes like Kata Containers, which run each pod inside a lightweight virtual machine with its own kernel, putting a hypervisor boundary between the tenant and the host.

In Kubernetes, these are wired in using a RuntimeClass, and pods opt in by setting runtimeClassName. In a multi-tenant cluster that opt-in needs to be enforced with admission control, otherwise a tenant can just leave the field out and get a standard container.

Trade-offs

These aren’t free, which is why they’re not the default.

• Overhead. Both use more memory and CPU per pod than runc, VM-based runtimes especially, which eats into the cost savings that made multi-tenancy attractive in the first place.
• Compatibility. Not everything behaves the same. When I looked at Kata options like --privileged, --net=host and --pid=host didn’t work as expected, and gVisor presents a quite different /proc to the workload. Those posts are from 2018 so things will have moved on, but it’s worth testing your workloads before committing.
• Node support. You need nodes (and on cloud, often specific instance types) that can run the runtime. Pairing this with dedicated node pools for sandboxed workloads is a common approach.

What sandboxed runtimes don’t fix

• The cluster-level risks. The tenant still shares the API server, the network and any operators in the cluster.
• The rest of the node. The kubelet, its credentials and the cluster’s DaemonSets are still running on the host, usually as standard containers. A sandbox makes it much harder to reach them, but doesn’t remove them.
• Sandbox escapes. gVisor and Kata have a much smaller attack surface than the full Linux kernel, but it’s not zero.