Resolved -
On July 21st, 2025, between 07:00 UTC and 09:45 UTC the API, Codespaces, Copilot, Issues, Package Registry, Pull Requests and Webhook services were degraded and experienced dropped requests and increased latency. At the peak of this incident (a 2 minute period around 07:00 UTC) error rates peaked at 11% and went down shortly after. Average web request load times rose to 1 second during this same time frame. After this period, traffic gradually recovered but error rate and latency remained slightly elevated until the end of the incident.
This incident was triggered by a kernel bug that caused a crash of some of our load balancers during a scheduled process after a kernel upgrade. In order to mitigate the incident, we halted the roll out of our upgrades, and rolled back the impacted instances. We are working to make sure the kernel version is fully removed from our fleet. As a precaution, we temporarily paused the scheduled process to prevent any unintended use in the affected kernel. We also tuned our alerting so we can more quickly detect and mitigate future instances where we experience a sudden drop in load-balancing capacity.
Jul 21, 09:48 UTC
Update -
API Requests and Codespaces are operating normally.
Jul 21, 09:47 UTC
Update -
Copilot is operating normally.
Jul 21, 09:45 UTC
Update -
Webhooks is operating normally.
Jul 21, 09:44 UTC
Update -
Mitigations have been applied and we are seeing recovery. We are continuing to closely monitor the situation to ensure complete recovery has been achieved.
Jul 21, 09:41 UTC
Update -
Issues is operating normally.
Jul 21, 09:34 UTC
Update -
Packages is operating normally.
Jul 21, 09:19 UTC
Update -
We are currently implementing mitigations for this issue.
Jul 21, 09:00 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
Jul 21, 08:48 UTC
Update -
We continue to investigate reports of degraded performance and intermittent timeouts across GitHub.com.
Jul 21, 08:27 UTC
Update -
Pull Requests is operating normally.
Jul 21, 08:10 UTC
Update -
We're continuing to investigate reports of degraded performance and intermitiant timeouts across GitHub.com.
Jul 21, 07:46 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Jul 21, 07:25 UTC
Update -
Packages is experiencing degraded performance. We are continuing to investigate.
Jul 21, 07:23 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Jul 21, 07:22 UTC
Update -
Codespaces is experiencing degraded performance. We are continuing to investigate.
Jul 21, 07:19 UTC
Investigating -
We are investigating reports of degraded performance for Issues and Webhooks
Jul 21, 07:15 UTC