RBAC & Operator Hygiene

Do tenants need API access at all?

The simplest way to avoid tenant RBAC problems is not to give tenants direct access to the Kubernetes API. If they deploy through a GitOps pipeline or a developer portal, a controller makes the changes on their behalf, there’s a nice audit trail, and the whole RBAC risk surface gets a lot smaller. It does move some of the problem onto the Git platform and the pipeline, so those become part of the tenant boundary.

Tenant RBAC

Where tenants do need API access:

• Use the built-in roles, not wildcards. The default admin and edit ClusterRoles, bound with a RoleBinding in the tenant’s namespace, are a much better starting point than a wildcard Role or a namespace-scoped binding to cluster-admin. The docs note that admin doesn’t allow write access to resource quotas or to the namespace object itself, which closes off the namespace relabelling trick.
• Know what the built-in roles include. They’re aggregated, so they pick up extra rules from any ClusterRole carrying the right aggregation label, and operators often add their CRDs that way. edit can also impersonate service accounts in its namespace and write NetworkPolicies, and on clusters originally created before 1.22 it can still write Endpoints.
• Audit for the dangerous ones. escalate, bind, impersonate, nodes/proxy, GET on pods/exec, CSR approval and controllerrevisions write are all worth searching for regularly. My eathar tool has some checks for this kind of thing (e.g. eathar rbac wildcardusers).
• Be careful with service account tokens. Turn off automountServiceAccountToken where it’s not needed, use a dedicated service account per workload, and look out for Helm charts that bind rights to the namespace’s default service account.

Operator hygiene

On the operator side, the aim is to understand what tenants can make each operator do, and then shrink it:

1. Pair up tenant-creatable custom resources with operator RBAC. For every operator, list the custom resource kinds a tenant can create and the rights of the service account that reconciles them. That pairing is the tenant’s effective privilege.
2. Find the operator-only resources. Operators often have internal CRDs they expect to create themselves (cert-manager’s Challenge is a good example). Make sure tenant RBAC, including wildcards and aggregated roles, can’t create them.
3. Cut operator rights back. Cluster-wide Secret read, rights to create RBAC objects, and permissions for sub-components you haven’t deployed are the common over-grants.
4. Scope or split operators. Operators that only watch specific namespaces, or a separate operator instance per tenant, limit what a confused deputy can reach.
5. Check references at admission. The Breaking the Bulkhead paper suggests an admission webhook that runs a SubjectAccessReview to check that the requesting user could access whatever their custom resource refers to.
6. Turn off programmable features you don’t need, like templating, CEL HTTP calls and policy callbacks, as these run tenant logic with operator rights.
7. Restrict operator egress with a network policy, so they can’t be tricked into sending their credentials to an arbitrary host.
8. Rotate operator credentials after a credential-theft CVE. Patching the operator doesn’t revoke a token that’s already been stolen. Baselining which namespaces, verbs and source IPs each operator service account normally uses in your audit logs makes it a lot easier to spot one being misused.

What this doesn’t fix

Least privilege can’t fully fix a confused deputy, because the operator really does need its permissions to do its job. What it does is limit how bad each bug turns out to be. It also doesn’t help with the information leakage that comes with any API access, or with operator bugs nobody has found yet, which is a good reason to be picky about how many cluster-wide operators a shared cluster runs.