Resolved -
On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues.
We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied.
We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.
May 12, 17:43 UTC
Update -
All services have fully recovered.
May 12, 17:43 UTC
Update -
CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.
May 12, 16:59 UTC
Update -
Webhooks have fully recovered. Continuing to work on recovery for the other services.
May 12, 16:29 UTC
Update -
Webhooks is operating normally.
May 12, 16:28 UTC
Update -
We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.
May 12, 16:18 UTC
Update -
Webhooks is experiencing degraded performance. We are continuing to investigate.
May 12, 15:44 UTC
Update -
We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.
May 12, 15:42 UTC
Update -
CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.
May 12, 15:13 UTC
Investigating -
We are investigating reports of degraded performance for CodeQL
May 12, 14:38 UTC