Runtime.Diaz

K8s Dynamic Admission Control: Mutating Admission Policies

Why it matters? | The big picture | Configuration | Use cases | User Workflows | Challenges | What's next?

Why it matters?

Kubernetes now provides native support for mutating resources during the admission phase as an alternative to mutating admission webhooks.

Introducing:

Important: With mutating admission policies added, the the mutating admission plugin order will become:

The big picture

Mutating Admission Policies was released as alpha in 1.32. The beta release is in progress and can be followed by the SIG API Machineery KEP 3962.

This new feature is a continuation of the previously released Validating Admission Policies (VAP) which has similarity in the API shape and configuration. I'll explore VAP in my next blog post.

Configuration

Find a lot more details in these resources: K8s docs | KEP 3962 | Source code | Kubecon NA 2024 Session | K8s CEL Docs | CEL Spec | External CEL macro assistance

Policy resources:

Note: I'll explore paramaterizing policies in the Validating Mutating Policies blog post.

Example MutatingAdmissionPolicy Resource

---
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: MutatingAdmissionPolicy
metadata:
  name: "add-required-namespace-labels"
spec:
  matchConstraints:
    resourceRules:
    - apiGroups:   [""]
      apiVersions: ["v1"]
      operations:  ["CREATE"]
      resources:   ["namespaces"]
  failurePolicy: Fail
  reinvocationPolicy: IfNeeded
  mutations:
    - patchType: "ApplyConfiguration"
      applyConfiguration:
        expression: >
          Object{
            metadata: Object.metadata{
              labels: Object.metadata.labels{
                type : "user-workload",
                systemid : object.metadata.name.split('-')[0],
              }
            }
          }

Example MutatingAdmissionPolicyBinding Resource

---
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: MutatingAdmissionPolicyBinding
metadata:
  name: "add-required-namespace-labels-binding"
spec:
  policyName: "add-required-namespace-labels"

Use Cases

The mutation requirements below are illustrative and were created to test MAP features against cluster resources.

Mutations requirements:

  1. Add the following namespace labels to ALL user namespaces:
    • systemid:
    • nstype: user
  2. Mutate (default) the container's imagePullPolicy to "Always" for pods with label platform: true only.
  3. Remove the resources.request.limits for ALL containers that include limits.

User workflows

Access a free kubernetes playground here: Fire up a ephemeral kubernetes mutating admission policies playground.

Apply the preexisting Mutating Admission Policies and Bindings to the cluster under the policy/ directory. Feel free to explore the policies for understanding.

> kubectl apply -f policy/
mutatingadmissionpolicybinding.admissionregistration.k8s.io/add-required-namespace-labels-binding created
mutatingadmissionpolicy.admissionregistration.k8s.io/add-required-namespace-labels created
mutatingadmissionpolicybinding.admissionregistration.k8s.io/image-pull-policy-binding created
mutatingadmissionpolicy.admissionregistration.k8s.io/image-pull-policy created
mutatingadmissionpolicybinding.admissionregistration.k8s.io/remove-resources-limits-binding created
mutatingadmissionpolicy.admissionregistration.k8s.io/remove-resources-limits created

Test mutation requirement 1

Create a namespace using this pattern: <systemid>-<service-name>-<environment>. For example: 54321-myservice-prod

❯ kubectl create ns 54321-myservice-prod
namespace/54321-myservice-prod created

❯ kubectl get ns 54321-myservice-prod -oyaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2025-01-14T01:06:52Z"
  labels:
    kubernetes.io/metadata.name: 54321-myservice-prod
    systemid: "54321" # mutated by the MAP 
    type: user-workload # mutated by the MAP
  name: 54321-myservice-prod
  resourceVersion: "886"
  uid: 323287ce-c16c-4e1e-ad14-d8bd2eec1413
spec:
  finalizers:
    - kubernetes
status:
  phase: Active

Test mutation requirement 2

Create two pods, one with the platform: true label, the other without the label.

❯ kubectl run mutate-image-pull --image nginx:alpine --labels=platform=true
pod/mutate-image-pull created
❯ kubectl run no-mutate-image-pull --image nginx:alpine
pod/no-mutate-image-pull created

Observe the running pod's output and verify the pod with the platform label was mutated to imagePullPolicy: Never.

❯ kubectl get pods -o yaml | yq '.items[].spec.containers'
- image: nginx:alpine
  imagePullPolicy: Never
  name: mutate-image-pull
  resources: {}
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-trxhw
      readOnly: true
- image: nginx:alpine
  imagePullPolicy: IfNotPresent
  name: no-mutate-image-pull
  resources: {}
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-877dm
      readOnly: true

Test mutation requirement 3

Explore and apply the multi-container-resource-pod.yaml manifest to the cluster found in the resources/ directory.

❯ kubectl apply -f resources/multi-container-resource-pod.yaml
pod/multi-container-resource-pod created

Observe and validate the container named c1's resources.limits.cpu field was removed.

❯ kubectl get pods multi-container-resource-pod -oyaml |yq '.spec.containers[].resources'
limits:
  memory: 128Mi
requests:
  cpu: 250m
  memory: 64Mi
limits:
  memory: 128Mi
requests:
  cpu: 250m
  memory: 64Mi
{}

Challenges

Challenge 1

I faced a few challenges getting started with the new feature while following the K8s docs.

Note: per their documentation it does seem that the error aligns to the ApplyConfiguration patchType statement

Apply configurations may not modify atomic structs, maps or arrays due to the risk of accidental deletion of values not included in the apply configuration.

Policy and binding

---
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: MutatingAdmissionPolicy
metadata:
  name: "sidecar-policy.example.com"
spec:
  matchConstraints:
    resourceRules:
    - apiGroups:   [""]
      apiVersions: ["v1"]
      operations:  ["CREATE"]
      resources:   ["pods"]
  failurePolicy: Fail
  reinvocationPolicy: IfNeeded
  mutations:
    - patchType: "ApplyConfiguration"
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              initContainers: [
                Object.spec.initContainers{
                  name: "mesh-proxy",
                  image: "mesh/proxy:v1.0.0",
                  args: ["proxy", "sidecar"],
                  restartPolicy: "Always"
                }
              ]
            }
          } 
---
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: MutatingAdmissionPolicyBinding
metadata:
  name: "sidecar-binding-test.example.com"
spec:
  policyName: "sidecar-policy.example.com"

$ kubectl apply -f pod.yaml
The pods "p1" is invalid: : policy 'sidecar-policy.example.com' with binding 'sidecar-binding-test.example.com' denied request: error applying patch: invalid ApplyConfiguration: may not mutate atomic arrays, maps or structs: .spec.initContainers[0].args

The way I did get to get their example to work is to use the JSONPatch patchType.

  mutations:
    - patchType: "JSONPatch"
      jsonPatch:
        expression: >
          [
            JSONPatch{
              op: "add", path: "/spec/initContainers/-",
              value: Object.spec.initContainers{
                name: "mesh-proxy",
                image: "mesh-proxy/v1.0.0",
                command: ["bin/sh", "-c"],
                restartPolicy: "Always"
              }
            }
          ]

Challenge 2

Implementing container-level security context mutations required complex CEL expressions to properly incorporate Pod-level security context requirements.

What's Next