What Is Multi-Tenancy?

When talking about Kubernetes multi-tenancy from a security standpoint, there can be different ways of framing what’s being discussed, for example this is the Kubernetes documentation site page on Multi-Tenancy. For the purposes of this site (and the talk) I came up with a definition of what I mean by it.

Essentially there are two variables that we’re considering :-

The first one is how much trust do we place in the individual tenants.

• Same team. Here this is a situation where multiple applications from a single team are run in a single Kubernetes cluster. As it’s the same team managing them, and the applications (hopefully) are developed and secured in the same way, we have quite a bit of trust between tenants. We still need controls to mitigate the risk of an external attacker compromising one of the applications and then trying to expand their access to other applications in the cluster, but we’re less concerned with malicious users trying to cross tenant boundaries or one application developed insecurely, leading to compromise of other higher security applications.

• Different team or environment. Here our tenants might have a different threat model or risk posture that means we have to place additional controls between tenants. This could be multiple different teams at the same organization who have different development practices and/or threat models, or it could be the idea of having development/test/production applications in the same cluster, where a development environment is likely to have less rigorous controls placed on it. Once we have more variability in threat models, additional controls are a good idea.

• External customers as tenants. Here we have essentially untrusted parties as tenants, who we need to treat as possibly actively hostile. There may be some controls from contracts that are in place, but tenants should be considered possibly malicious from a security perspective.

The second axis is more specific to Kubernetes and it focuses on how many rights each tenant has. This will inform the types of controls that are necessary and how easy it will be to isolate tenants.

• Application access. Here tenants and users can only see deployed applications, they have no direct access to the cluster or nodes. There are still some risks to consider based on application breakout attacks leading to container access, but it’s another layer of security that would need to be compromised before they were able to influence the cluster or containers directly.

• Container shell. Where tenants are given a shell in their application containers, generally for debugging/maintenance reason. This opens up attacks that can be executed from a container (e.g. container breakout attacks) and networking attacks on other tenants.

• Specified container image. In a Containers As A Service type deployment, tenants can effectively specify what image they want to run. Generally this will allow them tool access, and will also let them specify the software available to be used. It can also open up specific vectors where a malicious container image is used for breakouts. Lastly this might open up risks of compromise container registries depending on how they specify and deploy their container images.

• Custom manifests. Here the tenant can specify not only the container image to be used but other aspects of the deployment manifest. This opens up risks of a tenant giving their containers access to the underlying nodes, via things like privileged: true or mounting the whole node file-system into the container.

• Limited Kubernetes Namespace Access. In this setup tenants are provided with access to a namespace and direct Kubernetes API access to deploy and manage their workloads in that namespace. This configuration opens up Kubernetes RBAC risks as, if given the wrong permission, it might be possible for tenants to escalate their privileges to the cluster.

• Namespace As A Service. In this configuration tenants are provided with effectively full rights to their namespace via something like admin or edit rights. Here there are additional concerns like editing their namespace object to bypass security controls