Table of Contents
What Happened
On August 6, 2026 at 15:22 UTC, GitHub began investigating degraded performance in Actions. What followed was one of the more consequential CI/CD incidents in recent memory: a 10-hour-42-minute outage that wasn’t fully resolved until 02:04 UTC on August 7.
Most outages are an inconvenience — jobs run slow, then catch up. This one was different. Because of how it failed, some workflow-triggering events were dropped entirely and cannot be replayed automatically. If you push code or opened a pull request during the window and assumed CI ran, it may never have started at all. That’s the detail every team needs to act on.
Official incident record: githubstatus.com/incidents/qcvjkzcs7j74
The Blast Radius
This wasn’t confined to workflow runs. Over the course of the incident, the failure rippled across a surprising number of dependent services:
- Workflow runs failed to start, failed partway through, or sat queued until they timed out.
- The Actions REST API returned errors, and some workflows hit unexpected rate limiting.
- Both GitHub-hosted and self-hosted runners were affected — self-hosted runners saw errors and rate limiting during registration.
- Webhooks were throttled to roughly 15% at the low point, meaning the majority of push and pull request events never triggered a workflow.
- GitHub Pages, Copilot Code Review, Copilot Coding Agent, and GitHub Enterprise Importer all suffered collateral damage. Enterprise Importer migrations were deliberately paused as a precaution.
At its worst, job success rates dropped to 30–40%. The fact that Copilot’s code review and coding agent went down alongside Actions is a reminder of how much of GitHub’s newer surface area sits on top of the same execution plane.
Why This One Is Different: Events That Are Gone for Good
Here’s the technical heart of the incident, and the reason it deserves a closer look.
To protect the recovering system, GitHub throttled webhook delivery — the mechanism that turns a push or a pull request update into a triggered workflow run. During the throttle, only about 15% of webhooks were processed. The rest were not queued for later delivery. They were simply not processed.
GitHub’s own resolution note is blunt about the consequence:
“Some workflow-triggering events, including push and pull request events, were not processed during the incident and cannot be replayed automatically. Customers may need to repeat the triggering action by pushing a new commit, updating the pull request, or manually re-running the workflow where applicable.”
Read that carefully. This is not “your job was delayed.” This is “your job never ran, and GitHub cannot start it for you retroactively.”
See also: Mastering the Linux Command Line — Your Complete Free Training Guide
The dangerous part is the silence. A failed workflow shows a red X — you notice it. A workflow that never started shows nothing at all. A required status check that never runs can leave a PR looking clean, or leave a merge-to-main without the deploy pipeline that was supposed to follow it.
The Self-Hosted Runner Trap: Stuck-Idle ARC Pods
If you run Actions Runner Controller (ARC) on Kubernetes, there’s a second problem that outlived the main incident.
During the outage, runners were assigned jobs that were no longer valid, and some got stuck retrying jobs that no longer existed. Even after GitHub mitigated the core issue, a subset of ARC runner pods remained stuck in an idle state — they came back but never picked up new work.
GitHub’s guidance for affected users:
“Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.”
So if your self-hosted CI is quiet even though the incident is marked resolved, the runners themselves may need a manual kick:
# Identify runner pods that are stuck idle
kubectl get pods -n arc-runners
# Delete the stuck pods — ARC will recreate them automatically
kubectl delete pod <stuck-runner-pod> -n arc-runners
# Or, if many are affected, redeploy the ARC controller
kubectl rollout restart deployment <arc-controller> -n arc-systemsGitHub has said the next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism so these manual steps won’t be needed in the future. Until then, this remains a manual cleanup.
What You Need to Do Right Now
Because dropped events won’t replay themselves, recovery is on you. Here’s a concrete checklist to make sure nothing important slipped through the gap between 15:22 UTC Aug 6 and 02:04 UTC Aug 7.
1. Audit pushes and PRs made during the window
Any commit pushed or PR opened/updated in that ~11-hour window may have no corresponding workflow run.
# List commits pushed to main during the incident window (UTC)
git log --since="2026-08-06T15:22Z" --until="2026-08-07T02:04Z" \
--pretty=format:"%h %an %ad %s" --date=isoCross-reference these against the Actions tab. Any commit with no associated run needs attention.
2. Re-trigger missing workflows
For anything that didn’t run, force a fresh trigger:
- Push an empty commit to re-fire push-triggered workflows:
git commit --allow-empty -m "chore: re-trigger CI after GitHub Actions outage"
git push- Update the pull request (a new commit, or close/reopen) to re-fire PR-triggered workflows.
- Manually re-run via the Actions UI or CLI where a
workflow_dispatchtrigger exists:
gh workflow run <workflow-file>.yml --ref <branch>3. Check for “silent” merges
The most dangerous case: a PR that merged while its required checks never actually executed (if branch protection was bypassed, or the check was reported before the throttle). Review anything merged during the window and confirm its CI/CD pipeline — especially deploys — actually completed.
4. Verify self-hosted / ARC runners
If you use ARC, confirm no pods are stuck idle (see the kubectl steps above). For non-Kubernetes self-hosted runners, confirm they re-registered cleanly and are picking up jobs.
5. Confirm downstream deploys
If your deploy pipeline is triggered by a successful workflow, a dropped trigger means a deploy that never happened. Reconcile what’s actually running in production against what should have shipped.
The Broader Lesson: Your CI/CD Provider Is a Single Point of Failure
This incident is a useful, uncomfortable reminder that “green means shipped” is an assumption, not a guarantee. A few takeaways worth internalizing:
- “No run” is a failure mode, too. Most alerting watches for failed runs. Very few teams alert on missing runs. Consider monitoring that a workflow ran at all for critical branches — a dead man’s switch, not just a failure alarm.
- Don’t treat webhook delivery as guaranteed. Webhooks can be throttled or dropped under load. If a workflow trigger is business-critical, build a reconciliation path that can detect and replay gaps.
- Manual re-trigger paths matter. Every important workflow should have a
workflow_dispatchtrigger so you can fire it by hand when automation fails. - Know your ARC recovery runbook. If you self-host on Kubernetes, the
kubectl delete podrecovery step should already be in your incident playbook — not something you discover mid-outage.
Timeline at a Glance
| Time (UTC) | Status |
|---|---|
| Aug 6, 15:22 | Investigation begins — degraded Actions performance |
| Aug 6, 15:45 | Workflows failing to start / mid-run; Actions API errors |
| Aug 6, ~20:34 | Webhooks throttled to ~15%; job success ~65% (up from a 30–40% low) |
| Aug 6, 23:13 | Fixes deployed; running-workflow success back to ~99% |
| Aug 7, 00:06 | Degradation mitigated; monitoring begins |
| Aug 7, 00:59 | Reports of ARC runners slow to recover |
| Aug 7, 02:03 | Notice: dropped push/PR events cannot be auto-replayed |
| Aug 7, 02:04 | Incident resolved (~10h 42m total) |
A detailed root cause analysis is expected from GitHub and had not yet been published at the time of writing.
Were you affected by this outage? How did you detect the missing runs — and did any silent merges slip through? Share your recovery approach in the comments.
