GitHub Status

Dec 5, 2024

No incidents reported today.

Dec 4, 2024

Disruption with some GitHub services

Resolved - This incident has been resolved.
Dec 4, 19:27 UTC

Update - Pull Requests is operating normally.
Dec 4, 19:26 UTC

Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:21 UTC

Update - Issues is operating normally.
Dec 4, 19:20 UTC

Update - API Requests is operating normally.
Dec 4, 19:18 UTC

Update - Webhooks is operating normally.
Dec 4, 19:17 UTC

Update - We have identified the cause of timeouts impacting users across multiple services. This change was rolled back and we are seeing recovery. We will continue to monitor for complete recovery.
Dec 4, 19:11 UTC

Update - Issues is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:07 UTC

Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:05 UTC

Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:05 UTC

Investigating - We are currently investigating this issue.
Dec 4, 18:58 UTC

Dec 3, 2024

[Retroactive] Incident with Pull Requests

Resolved - On December 3rd, between 23:29 and 23:43 UTC, Pull Requests experienced a brief outage and teams have confirmed the issue to be resolved. Due to brevity of incident it was not publicly statused at the time however an RCA will be conducted and shared in due course.
Dec 3, 23:30 UTC

Incident with Pull Requests and API Requests

Resolved - On December 3, 2024, between 19:35 UTC and 20:05 UTC API requests, Actions, Pull Requests and Issues were degraded. Web and API requests for Pull Requests experienced a 3.5% error rate and Issues had a 1.2% error rate. The highest impact was for users who experienced errors while creating and commenting on Pull Requests and Issues. Actions had a 3.3% error rate in jobs and delays on some updates during this time.

This was due to an erroneous database credential change impacting write access to Issues and Pull Requests data. We mitigated the incident by reverting the credential change at 19:52 UTC. We continued to monitor service recovery before resolving the incident at 20:05 UTC.

There are a few improvements we are making in response to this. We are investing in safe guards to the change management process in order to prevent erroneous database credential changes. Additionally, the initial rollback attempt was unsuccessful which led to a longer time to mitigate. We were able to revert through an alternative method and are updating our playbooks to document this mitigation strategy.
Dec 3, 20:05 UTC

Update - Pull Requests is operating normally.
Dec 3, 20:05 UTC

Update - Actions is operating normally.
Dec 3, 20:04 UTC

Update - API Requests is operating normally.
Dec 3, 20:02 UTC

Update - We have taken mitigating actions and are starting to see recovery but are continuing to monitor and ensure full recovery. Some users may still see errors.
Dec 3, 19:59 UTC

Update - Some users will experience problems with certain features of pull requests, actions, issues and other areas. We are aware of the issue, know the cause, and are working on a mitigation.
Dec 3, 19:54 UTC

Investigating - We are investigating reports of degraded performance for API Requests, Actions and Pull Requests
Dec 3, 19:48 UTC

Disruption with some GitHub services

Resolved - This incident has been resolved.
Dec 3, 04:39 UTC

Update - We saw a recurrence of the large hosted runner incident (https://www.githubstatus.com/incidents/qq1m7mqcl6zk) from 12/1/2024. We've applied the same mitigation and see improvements. We will continue to work on a long term solution.
Dec 3, 04:38 UTC

Update - We are investigating reports of degraded performance for Hosted Runners
Dec 3, 04:16 UTC

Investigating - We are currently investigating this issue.
Dec 3, 04:11 UTC

Dec 2, 2024

Disruption with some GitHub services

Resolved - Between Dec 1 12:20 UTC and Dec 2 1:05 UTC, availability of large hosted runners for Actions was degraded due to failures in background VM provisioning jobs. Users would see workflows queued waiting for a runner. On average, 8% of all workflows requiring large runners over the incident time were affected, peaking at 37.5% of requests. There were also lower levels of intermittent queuing on Dec 1 beginning around 3:00 UTC. Standard and Mac runners were not affected.

The job failures were caused by timeouts to a dependent service in the VM provisioning flow and gaps in the jobs’ resilience to those timeouts. The incident was mitigated by circumventing the dependency as it was not in the critical path of VM provisioning.

There are a few immediate improvements we are making in response to this. We are addressing the causes of the failed calls to improve the availability of calls to that backend service. Even with that impact, the critical flow of large VM provisioning should not have been impacted, so we are improving the client behavior to fail fast and circuit break non-critical calls. Finally the alerting for this service was not adequate in this particular scenario to ensure fast response by our team. We are improving our automated detection from this to reduce our time to detection and mitigation of issues like this one in the future.
Dec 2, 01:05 UTC

Update - We've applied a mitigation to fix the issues with large runner jobs processing. We are seeing improvements in telemetry and are monitoring for full recovery.
Dec 2, 00:57 UTC

Update - We continue to investigate large hosted runners not picking up jobs.
Dec 2, 00:14 UTC

Update - We continue to investigate issues with large runners.
Dec 1, 23:43 UTC

Update - We're seeing issues related to large runners not picking up jobs and are investigating.
Dec 1, 23:24 UTC

Investigating - We are currently investigating this issue.
Dec 1, 23:18 UTC

Dec 1, 2024

Nov 30, 2024

No incidents reported.

Nov 29, 2024

No incidents reported.

Nov 28, 2024

Incident with Codespaces

Resolved - This incident has been resolved.
Nov 28, 07:01 UTC

Update - We identified the issue and applied a mitigation, resulting in the cessation of timeouts. While we are considering this incident resolved for now, we are continuing to investigate the root cause and plan to implement a permanent fix. Updates will follow as we progress.
Nov 28, 07:00 UTC

Update - We are investigating issues with timeouts in some requests in Codespaces. We will update you on mitigation progress.
Nov 28, 06:36 UTC

Investigating - We are investigating reports of degraded performance for Codespaces
Nov 28, 06:27 UTC

Incident with Sporadic Timeouts in Codespaces

Resolved - This incident has been resolved.
Nov 28, 05:11 UTC

Update - We identified and addressed failures in two proxy servers and applied mitigation. Since then, timeouts have ceased, and we are considering the incident resolved. We will continue to monitor the situation closely and provide updates if any changes occur.
Nov 28, 05:10 UTC

Update - We are investigating some network proxy issues that may be contributing to the timeouts in a small percentage of requests in Codespaces. We will continue to investigate.
Nov 28, 04:34 UTC

Update - We are continuing to investigate issues with timeouts in a small percentage of requests in Codespaces. We will update you on mitigation progress.
Nov 28, 04:03 UTC

Update - We are investigating issues with timeouts in some requests in Codespaces. Some users may not be able to connect to their Codespaces at this time. We will update you on mitigation progress.
Nov 28, 03:32 UTC

Investigating - We are investigating reports of degraded performance for Codespaces
Nov 28, 03:29 UTC

Nov 27, 2024

No incidents reported.

Nov 26, 2024

No incidents reported.

Nov 25, 2024

Disruption with GitHub Search

Resolved - Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.
Nov 25, 15:25 UTC

Update - Search is now operating normally. We are declaring this issue resolved.
Nov 25, 15:24 UTC

Update - We are now observing signs of complete recovery in search. We will continue to monitor and assess.
Nov 25, 15:10 UTC

Update - We're observing signs of recovery in search, we will continue to monitor.

Next update within 15 minutes.

Nov 25, 14:40 UTC

Update - We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.

Code and issues search continue to be available.

The team is investigating, next update in 30m.
Nov 25, 14:10 UTC

Update - We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.
Nov 25, 13:58 UTC

Investigating - We are currently investigating this issue.
Nov 25, 13:57 UTC

Disruption with some GitHub services

Resolved - On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.

We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.
Nov 25, 12:17 UTC

Update - This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.
Nov 25, 12:17 UTC

Update - The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.

All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.
Nov 25, 11:44 UTC

Update - Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.
Nov 25, 11:00 UTC

Investigating - We are currently investigating this issue.
Nov 25, 10:51 UTC

Nov 24, 2024

No incidents reported.

Nov 23, 2024

No incidents reported.

Nov 22, 2024

No incidents reported.

Nov 21, 2024

Repository searches not working for some users

Resolved - On November 21, 2024, between 14:30 UTC and 15:53 UTC search services at GitHub were degraded and CPU load on some nodes hit 100%. On average, the error rate was 22 requests/second and peaked at 83 requests/second. During this incident Enterprise Profile pages were slow to load and searches may have returned low quality results.

The CPU load was mitigated by redeploying portions of our web infrastructure.

We are still working to identify the cause of the increase in CPU usage and are improving our observability tooling to better expose the cause of an incident like this in the future.
Nov 21, 16:48 UTC

Update - We are seeing recovery across all searches. The team continues to closely monitor our search system and is working to fully mitigate the cause of the problems.
Nov 21, 16:04 UTC

Update - Users will notice that loading an organization profile will sometimes not work. Additionally, the site-wide search is affected, too.
This issue does not affect code or issues and pull requests searches.
Nov 21, 15:33 UTC

Investigating - We are currently investigating this issue.
Nov 21, 15:30 UTC

About This Site