Policy as Code for Kubernetes Gatekeeping

Every platform team has a wiki page titled 'Kubernetes Standards' that nobody reads. Developers still push Deployments with :latest tags, missing resource requests, and containers running as root. Policy as code moves those standards from documentation into admission control-where violations fail at apply time, not in a 2 AM incident.

Gatekeeper (OPA) and Kyverno are the two dominant choices for Kubernetes policy. Both work. The decision is less about syntax and more about who writes policies, how fast you need feedback in CI, and whether you want mutate-as-well-as-validate semantics.

Admission Control in the Request Path

When kubectl or a GitOps controller applies a manifest, the API server runs it through admission webhooks before persistence. Validating webhooks reject bad configs; mutating webhooks can inject defaults.

kubectl apply / Argo CD sync
         │
         ▼
   Kubernetes API Server
         │
         ├── Mutating webhooks (defaults, sidecars)
         │
         └── Validating webhooks (Gatekeeper / Kyverno)
                 │
                 ├── ALLOW → etcd
                 └── DENY  → error to user / OutOfSync

Policies here are your last line of defense. CI scanning (Checkov, Kubesec, conftest) should catch issues earlier, but admission control catches everything-including kubectl edits and Helm charts from vendors.

Gatekeeper vs Kyverno: A Practical Comparison

We run Gatekeeper at Edmingle because platform engineers already use OPA for Terraform and API authorization. For teams without Rego experience, Kyverno's YAML policies ship faster and app developers can read them.

Policies That Actually Matter in Production

Start with a small set of high-impact constraints. A hundred policies in audit mode creates alert fatigue; five enforced policies change behavior.

• Require resource requests and limits on all containers.
• Deny privileged containers and hostPath volumes.
• Block images not from approved registries (ECR allowlist).
• Require standard labels: app, team, environment, cost-center.
• Deny LoadBalancer Services in namespaces without annotation approval.
• Require readOnlyRootFilesystem where the app supports it.

# Kyverno: require resource requests
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "CPU and memory requests are required"
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    memory: "?*"
                    cpu: "?*"

# Gatekeeper ConstraintTemplate (Rego excerpt concept)
# require_labels constraint denies Deployments missing team label
package k8srequiredlabels

violation[{"msg": msg}] {
  provided := {label | input.review.object.metadata.labels[label]}
  required := {"team", "environment", "app"}
  missing := required - provided
  count(missing) > 0
  msg := sprintf("Missing labels: %v", [missing])
}

Audit Before Enforce

Flipping enforcement on day one blocks deploys and makes you the bottleneck. Run every new policy in audit mode for at least two weeks. Export violations to Grafana or CloudWatch and share weekly reports with team leads.

Week 1-2: audit mode → collect violations per team
Week 3:   fix top offenders + update golden paths / Helm charts
Week 4:   enforce mode with documented exception process
Ongoing:  exception CRD or namespace annotation with expiry date

Maintain a policy exception workflow: temporary namespace annotation, ticket reference, auto-expire after 30 days. Permanent exceptions become policy debt-review quarterly.

Integrating Policy into CI and GitOps

Admission control alone frustrates developers who discover violations at sync time. Shift left with conftest or kyverno CLI in pull request checks against rendered manifests.

# GitHub Actions: conftest on rendered Helm
- name: Policy check
  run: |
    helm template api ./charts/api -f values-prod.yaml > rendered.yaml
    conftest test rendered.yaml -p policies/k8s

Argo CD users should wire policy violations into sync hooks or use OPA/Gatekeeper audit CRDs synced to Slack. The goal is the same error in CI that admission would produce in prod.

Engineering Takeaways

• Policy as code replaces unenforced wiki standards with admission-time rejection.
• Gatekeeper suits teams already invested in OPA; Kyverno lowers the bar for Kubernetes-native YAML policies.
• Start with five high-impact policies-not fifty audit-only rules nobody fixes.
• Always run audit mode before enforce; measure violations per team before blocking deploys.
• Shift left with CI policy checks so developers see failures in PRs, not at Argo CD sync.
• Exception workflows with expiry dates prevent permanent policy bypass sprawl.
• Combine validate (deny bad) with mutate (inject defaults) to reduce developer friction.

Trade-offs I Would Consider

Option: Gatekeeper with Rego policies

• Pros: Reusable OPA knowledge across K8s, Terraform, and APIs; mature ecosystem; strong audit tooling.
• Cons: Rego learning curve; app teams rarely contribute; debugging violations requires platform expertise.

Option: Kyverno with YAML policies

• Pros: Readable policies, native mutate/generate, faster onboarding for K8s engineers.
• Cons: K8s-only; complex logic can become verbose; less portable to non-K8s policy use cases.

What I Would Do in Production

Deploy Gatekeeper or Kyverno in HA on the management cluster; pin versions and test upgrades in staging first.

Maintain a policies repo versioned alongside cluster config; Argo CD syncs ConstraintTemplates before Constraints.

Publish a 'golden Deployment' Helm chart that passes all policies out of the box-make the compliant path the easy path.

Dashboard violation counts by team and namespace; escalate repeat offenders to engineering managers with data.

Include policy checks in your platform onboarding doc: new services scaffold from the golden chart, not copy-paste from Stack Overflow.

Test policies against your own platform namespaces-kube-system exemptions should be explicit, not accidental.

FAQ

Q: Will policies slow down API server?

With reasonable match criteria and caching, impact is usually negligible. Avoid overly broad matches (every resource type cluster-wide) and load-test admission latency during policy rollouts.

Q: How do we handle emergency hotfixes?

Break-glass namespace with relaxed policies, MFA-gated access, and mandatory post-incident review. Do not disable Gatekeeper cluster-wide-it will not get re-enabled under pressure.

Q: Can policies enforce network policies automatically?

Kyverno generate rules can create default NetworkPolicies for new namespaces. Gatekeeper equivalent exists via mutation or separate automation. Policy should not replace thoughtful network segmentation design.

Q: What about third-party Helm charts?

Vendor charts often fail your policies. Fork, patch, or wrap with a platform-maintained values layer. CI conftest on vendor chart renders before merge.

Q: Should prod and non-prod share policies?

Same baseline everywhere; prod adds stricter rules (no latest tag, no LoadBalancer). Use policy parameters or separate Constraint instances per environment.