A CI identity that can only dry-run

← Writing

I want CI to check my releases against the real cluster before they’re merged, without being able to deploy them. Kubernetes permissions don’t support this directly, so this post shows how I set it up.

01

Checking without deploying

Kubernetes has a built-in way to test a change: a server-side dry-run.

kubectl apply --dry-run=server -f release.yaml

The API server treats this like a real request. It validates the schema, fills in defaults, and runs the admission webhooks. But it doesn’t save anything. If the release has a problem, the dry-run catches most of the errors a real deploy would hit.

That’s what I want CI to do on every pull request.

02

RBAC can’t tell a test from the real thing

Kubernetes decides what an account may do with RBAC. An RBAC rule is basically a list: this account can create, update or delete these resources in this namespace.

A dry-run is still a create (or an update). There’s no “dry-run” verb, and no field in an RBAC rule for it. So if CI is allowed to dry-run a Deployment, it’s also allowed to create one for real. RBAC has no way to allow only dry-runs.

03

A second check after RBAC

After RBAC allows a request, the API server runs admission checks on it. Admission can see who sent the request and whether it’s a dry-run.

Since Kubernetes 1.30 you can write admission rules as plain Kubernetes objects, called ValidatingAdmissionPolicy. You apply them with kubectl like anything else. The API server evaluates them itself, so there’s no extra service to run.

So the setup is: give CI the same RBAC as the deployer, then add a policy that says “requests from CI must be dry-runs”.

It takes two objects. The policy holds the rule, and the binding switches it on and says what to do when a request breaks it (agent-rbac.yaml):

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: jenkins-ci-dry-run-only
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["*"]
        apiVersions: ["*"]
        operations: ["CREATE", "UPDATE", "DELETE"]
        resources: ["*/*"]
  matchConditions:
    - name: jenkins-ci-requests-only
      expression: request.userInfo.username == 'system:serviceaccount:jenkins:jenkins-ci'
  validations:
    - expression: request.dryRun == true
      message: jenkins-ci may only send dry-run requests
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: jenkins-ci-dry-run-only
spec:
  policyName: jenkins-ci-dry-run-only
  validationActions: [Deny]

matchConditions makes the rule apply only to the CI service account. validations requires request.dryRun == true. The rest means create, update and delete on every resource.

04

Making sure it actually blocks

Some mistakes in the policy won’t show any error. CI just keeps its write access. So every time I run setup, it tries a real write as CI and expects it to be denied (environment/bin/up):

if denial="$(kubectl -n "${NAMESPACE}" create configmap delivery-probe \
  --from-literal=probe=1 \
  --as=system:serviceaccount:jenkins:jenkins-ci 2>&1)"; then
  kubectl -n "${NAMESPACE}" delete configmap delivery-probe >/dev/null
  fail 'jenkins-ci can write to the cluster'
fi

if [[ "${denial}" != *jenkins-ci-dry-run-only* ]]; then
  fail "unexpected CI denial: ${denial}"
fi

The first check fails setup if the write goes through.

The second check makes sure the policy is what denied it. A write can also fail for other reasons, for example if the namespace doesn’t exist yet. Without this check, that failure would pass as “CI is locked down”.

One thing the probe can’t catch: the policy finds CI by its username, and the probe uses the same hardcoded name. If CI moves to a different service account, the probe keeps testing the old one and passes. So the policy and the probe have to be updated together.

In the end CI has the same permissions as the deployer, but every real write from it gets denied, and setup fails if that stops being true. It can still read ConfigMaps and pod logs in those namespaces, which is fine for my lab setup.