Get email notifications whenever GitHub creates, updates or resolves an incident.
Get text message notifications whenever GitHub creates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge our Privacy Policy. In addition, you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Get webhook notifications whenever GitHub creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
About This Site
For the status of GitHub Enterprise Cloud - EU, please visit: eu.githubstatus.com
Git Operations
?
Operational
Webhooks
?
Operational
Visit www.githubstatus.com for more information
Operational
API Requests
?
Operational
Issues
?
Operational
Pull Requests
?
Operational
Actions
?
Operational
Packages
?
Operational
Pages
?
Operational
Codespaces
?
Operational
Copilot
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Dec 3, 2024
Resolved -
This incident has been resolved.
Dec 3, 04:39 UTC
Dec 3, 04:39 UTC
Update -
We saw a recurrence of the large hosted runner incident (https://www.githubstatus.com/incidents/qq1m7mqcl6zk) from 12/1/2024. We've applied the same mitigation and see improvements. We will continue to work on a long term solution.
Dec 3, 04:38 UTC
Dec 3, 04:38 UTC
Update -
We are investigating reports of degraded performance for Hosted Runners
Dec 3, 04:16 UTC
Dec 3, 04:16 UTC
Investigating -
We are currently investigating this issue.
Dec 3, 04:11 UTC
Dec 3, 04:11 UTC
Dec 2, 2024
Resolved -
Between Dec 1 12:20 UTC and Dec 2 1:05 UTC, availability of large hosted runners for Actions was degraded due to failures in background VM provisioning jobs. Users would see workflows queued waiting for a runner. On average, 8% of all workflows requiring large runners over the incident time were affected, peaking at 37.5% of requests. There were also lower levels of intermittent queuing on Dec 1 beginning around 3:00 UTC. Standard and Mac runners were not affected.
The job failures were caused by timeouts to a dependent service in the VM provisioning flow and gaps in the jobs’ resilience to those timeouts. The incident was mitigated by circumventing the dependency as it was not in the critical path of VM provisioning.
There are a few immediate improvements we are making in response to this. We are addressing the causes of the failed calls to improve the availability of calls to that backend service. Even with that impact, the critical flow of large VM provisioning should not have been impacted, so we are improving the client behavior to fail fast and circuit break non-critical calls. Finally the alerting for this service was not adequate in this particular scenario to ensure fast response by our team. We are improving our automated detection from this to reduce our time to detection and mitigation of issues like this one in the future.
Dec 2, 01:05 UTC
The job failures were caused by timeouts to a dependent service in the VM provisioning flow and gaps in the jobs’ resilience to those timeouts. The incident was mitigated by circumventing the dependency as it was not in the critical path of VM provisioning.
There are a few immediate improvements we are making in response to this. We are addressing the causes of the failed calls to improve the availability of calls to that backend service. Even with that impact, the critical flow of large VM provisioning should not have been impacted, so we are improving the client behavior to fail fast and circuit break non-critical calls. Finally the alerting for this service was not adequate in this particular scenario to ensure fast response by our team. We are improving our automated detection from this to reduce our time to detection and mitigation of issues like this one in the future.
Dec 2, 01:05 UTC
Update -
We've applied a mitigation to fix the issues with large runner jobs processing. We are seeing improvements in telemetry and are monitoring for full recovery.
Dec 2, 00:57 UTC
Dec 2, 00:57 UTC
Update -
We continue to investigate large hosted runners not picking up jobs.
Dec 2, 00:14 UTC
Dec 2, 00:14 UTC
Update -
We continue to investigate issues with large runners.
Dec 1, 23:43 UTC
Dec 1, 23:43 UTC
Update -
We're seeing issues related to large runners not picking up jobs and are investigating.
Dec 1, 23:24 UTC
Dec 1, 23:24 UTC
Investigating -
We are currently investigating this issue.
Dec 1, 23:18 UTC
Dec 1, 23:18 UTC
Dec 1, 2024
Nov 30, 2024
No incidents reported.
Nov 29, 2024
No incidents reported.
Nov 28, 2024
Resolved -
This incident has been resolved.
Nov 28, 07:01 UTC
Nov 28, 07:01 UTC
Update -
We identified the issue and applied a mitigation, resulting in the cessation of timeouts. While we are considering this incident resolved for now, we are continuing to investigate the root cause and plan to implement a permanent fix. Updates will follow as we progress.
Nov 28, 07:00 UTC
Nov 28, 07:00 UTC
Update -
We are investigating issues with timeouts in some requests in Codespaces. We will update you on mitigation progress.
Nov 28, 06:36 UTC
Nov 28, 06:36 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Nov 28, 06:27 UTC
Nov 28, 06:27 UTC
Resolved -
This incident has been resolved.
Nov 28, 05:11 UTC
Nov 28, 05:11 UTC
Update -
We identified and addressed failures in two proxy servers and applied mitigation. Since then, timeouts have ceased, and we are considering the incident resolved. We will continue to monitor the situation closely and provide updates if any changes occur.
Nov 28, 05:10 UTC
Nov 28, 05:10 UTC
Update -
We are investigating some network proxy issues that may be contributing to the timeouts in a small percentage of requests in Codespaces. We will continue to investigate.
Nov 28, 04:34 UTC
Nov 28, 04:34 UTC
Update -
We are continuing to investigate issues with timeouts in a small percentage of requests in Codespaces. We will update you on mitigation progress.
Nov 28, 04:03 UTC
Nov 28, 04:03 UTC
Update -
We are investigating issues with timeouts in some requests in Codespaces. Some users may not be able to connect to their Codespaces at this time. We will update you on mitigation progress.
Nov 28, 03:32 UTC
Nov 28, 03:32 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Nov 28, 03:29 UTC
Nov 28, 03:29 UTC
Nov 27, 2024
No incidents reported.
Nov 26, 2024
No incidents reported.
Nov 25, 2024
Resolved -
Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.
Nov 25, 15:25 UTC
Nov 25, 15:25 UTC
Update -
Search is now operating normally. We are declaring this issue resolved.
Nov 25, 15:24 UTC
Nov 25, 15:24 UTC
Update -
We are now observing signs of complete recovery in search. We will continue to monitor and assess.
Nov 25, 15:10 UTC
Nov 25, 15:10 UTC
Update -
We're observing signs of recovery in search, we will continue to monitor.
Next update within 15 minutes.
Nov 25, 14:40 UTC
Next update within 15 minutes.
Nov 25, 14:40 UTC
Update -
We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.
Code and issues search continue to be available.
The team is investigating, next update in 30m.
Nov 25, 14:10 UTC
Code and issues search continue to be available.
The team is investigating, next update in 30m.
Nov 25, 14:10 UTC
Update -
We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.
Nov 25, 13:58 UTC
Nov 25, 13:58 UTC
Investigating -
We are currently investigating this issue.
Nov 25, 13:57 UTC
Nov 25, 13:57 UTC
Resolved -
On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.
We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.
Nov 25, 12:17 UTC
We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.
Nov 25, 12:17 UTC
Update -
This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.
Nov 25, 12:17 UTC
Nov 25, 12:17 UTC
Update -
The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.
All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.
Nov 25, 11:44 UTC
All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.
Nov 25, 11:44 UTC
Update -
Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.
Nov 25, 11:00 UTC
Nov 25, 11:00 UTC
Investigating -
We are currently investigating this issue.
Nov 25, 10:51 UTC
Nov 25, 10:51 UTC
Nov 24, 2024
No incidents reported.
Nov 23, 2024
No incidents reported.
Nov 22, 2024
No incidents reported.
Nov 21, 2024
Resolved -
On November 21, 2024, between 14:30 UTC and 15:53 UTC search services at GitHub were degraded and CPU load on some nodes hit 100%. On average, the error rate was 22 requests/second and peaked at 83 requests/second. During this incident Enterprise Profile pages were slow to load and searches may have returned low quality results.
The CPU load was mitigated by redeploying portions of our web infrastructure.
We are still working to identify the cause of the increase in CPU usage and are improving our observability tooling to better expose the cause of an incident like this in the future.
Nov 21, 16:48 UTC
The CPU load was mitigated by redeploying portions of our web infrastructure.
We are still working to identify the cause of the increase in CPU usage and are improving our observability tooling to better expose the cause of an incident like this in the future.
Nov 21, 16:48 UTC
Update -
We are seeing recovery across all searches. The team continues to closely monitor our search system and is working to fully mitigate the cause of the problems.
Nov 21, 16:04 UTC
Nov 21, 16:04 UTC
Update -
Users will notice that loading an organization profile will sometimes not work. Additionally, the site-wide search is affected, too.
This issue does not affect code or issues and pull requests searches.
Nov 21, 15:33 UTC
This issue does not affect code or issues and pull requests searches.
Nov 21, 15:33 UTC
Investigating -
We are currently investigating this issue.
Nov 21, 15:30 UTC
Nov 21, 15:30 UTC
Nov 20, 2024
No incidents reported.
Nov 19, 2024
Resolved -
This incident has been resolved.
Nov 19, 12:03 UTC
Nov 19, 12:03 UTC
Update -
We have resolved the issue but are waiting for queues to catch up.
Nov 19, 11:56 UTC
Nov 19, 11:56 UTC
Update -
We're investigating an issue with two-factor authentication in the GitHub mobile app
Nov 19, 11:38 UTC
Nov 19, 11:38 UTC
Investigating -
We are currently investigating this issue.
Nov 19, 11:36 UTC
Nov 19, 11:36 UTC
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe