Automating Runbooks with GitHub Actions

Runbooks that live in Notion and execute in someone's terminal do not scale with on-call rotation. The engineer at 3 AM should not need to remember whether staging Redis lives in cluster A or B. Automating runbooks turns tribal knowledge into workflow_dispatch buttons-with audit logs.

GitHub Actions is not just CI. With OIDC to AWS, environment protection rules, and reusable workflows, it is a credible runbook execution engine for platform teams already standardized on GitHub.

What Belongs in an Automated Runbook

Not every runbook should be a workflow. Automate procedures that are frequent, well-defined, and low-judgment. Keep human-in-the-loop for anything that deletes data or crosses security boundaries without approval.

Architecture: Reusable Workflows and Environments

Structure runbooks as reusable workflows called by thin entry workflows. Platform team maintains the reusable layer; service teams get parameterized triggers.

Engineer clicks workflow_dispatch
         │
         ▼
┌─────────────────────┐
│  runbook-restart.yml │  (entry: inputs for service, env)
└──────────┬──────────┘
           │ uses
           ▼
┌─────────────────────┐
│  .github/reusable/   │
│  k8s-rollout.yml     │  (OIDC → EKS, kubectl rollout restart)
└─────────────────────┘
           │
           ▼
   GitHub Environment: production
   (required reviewers, wait timer)

# .github/workflows/runbook-rollout-restart.yml
name: Runbook - Rollout Restart
on:
  workflow_dispatch:
    inputs:
      service:
        description: Deployment name
        required: true
      namespace:
        description: K8s namespace
        default: production

jobs:
  restart:
    uses: ./.github/workflows/reusable-k8s-rollout-restart.yml
    with:
      service: ${{ inputs.service }}
      namespace: ${{ inputs.namespace }}
    secrets: inherit

OIDC to AWS Without Long-Lived Keys

Store no AWS access keys in GitHub secrets. Use OIDC federation so each workflow run assumes an IAM role scoped by repository, environment, and branch.

# Reusable job fragment
permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/gh-runbook-prod
      aws-region: ap-south-1

  - run: aws eks update-kubeconfig --name prod-cluster
  - run: kubectl rollout restart deployment/${{ inputs.service }} -n ${{ inputs.namespace }}

IAM trust policies should restrict sub claims: repo, environment, and optionally workflow name. A runbook role should not double as your CI deploy role.

Safety Rails: Approvals, Idempotency, and Slack

GitHub Environments named staging and production with required reviewers turn workflow_dispatch into a two-person rule for dangerous ops.

• Wait timer on production environment (5-10 min) to catch mis-clicks.
• Idempotent steps: check current state before mutate; no-op if already desired.
• Post results to Slack with actor, inputs, workflow run URL, and outcome.
• Auto-open incident ticket on failure via webhook.
• Concurrency groups per runbook + target to prevent overlapping restarts.

# Slack notification step
- name: Notify Slack
  if: always()
  run: |
    curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
      -d "{\"text\": \"Runbook ${{ github.workflow }} by ${{ github.actor }}: ${{ job.status }}\"}"

Testing Runbooks Like Code

Runbooks rot silently. Schedule weekly dry-run workflows against staging that execute read-only verification steps and optional mutate on ephemeral namespaces.

Version runbooks in Git. Require PR review from platform team. Link each workflow to the Notion runbook section it replaces-eventually delete the Notion steps that duplicate the workflow.

If the automated runbook and the wiki disagree, the wiki is wrong. Delete or update the wiki the same week you merge the workflow.

Engineering Takeaways

• Automate frequent, deterministic runbooks-not every procedure deserves a workflow.
• Reusable workflows plus GitHub Environments centralize safety rails and reduce copy-paste.
• OIDC to AWS eliminates long-lived keys; scope IAM roles per runbook and environment.
• Production runbooks need required reviewers, wait timers, and Slack audit trails.
• Idempotency checks prevent double-execution damage during incident panic.
• Test runbooks on staging on a schedule; treat workflow YAML like application code.
• When automation and wiki diverge, fix or delete the wiki-automation is source of truth.

Trade-offs I Would Consider

Option: Manual runbooks in wiki + kubectl

• Pros: Flexible for novel incidents, no workflow maintenance, works offline.
• Cons: Tribal knowledge, inconsistent execution, no audit trail, slow at 3 AM.

Option: GitHub Actions runbooks with environment gates

• Pros: Auditable, repeatable, self-service for engineers, integrates with existing GitHub access.
• Cons: Upfront authoring cost, GitHub outage blocks execution, not ideal for sub-minute emergencies.

What I Would Do in Production

Inventory top ten on-call procedures by frequency; automate the top three within one quarter.

Create a runbooks repo or .github/workflows/runbooks/ directory with naming convention runbook-<action>.yml.

Map each production workflow to a GitHub Environment with required reviewers from platform + service owner.

Use OIDC roles with least privilege: restart-runbook role cannot delete namespaces or modify RDS.

Post every execution to #platform-ops Slack with link to workflow run.

Quarterly review: disable workflows unused for 90 days, update inputs for renamed services.

FAQ

Q: Why GitHub Actions over Rundeck or Ansible Tower?

If your org already lives in GitHub, Actions avoids another tool, another auth system, and another audit log. Dedicated runbook platforms win when non-engineers execute procedures.

Q: What about emergencies needing sub-60-second response?

Keep a one-line kubectl alias runbook for total platform outage. Automate everything else. Not every path needs a workflow.

Q: How do we prevent runbook abuse?

Environment protection rules, IAM role scoping, and CloudTrail for AWS API calls. Alert on production workflow runs outside incident windows.

Q: Can runbooks call PagerDuty?

Yes-trigger workflows from PagerDuty webhooks for known alert types, or post back to PagerDuty when runbook completes to auto-resolve.

Q: Should developers run production runbooks?

With environment approval from on-call platform engineer, yes for scoped actions like rollout restart. Destructive ops stay platform-only.