GitHub Status

Mar 3, 2025

Incident with Issues, Git Operations and API Requests

Resolved - This incident has been resolved.
Mar 3, 05:31 UTC

Update - We have seen recovery across our services and impact is mitigated.
Mar 3, 05:30 UTC

Update - Git Operations is operating normally.
Mar 3, 05:20 UTC

Update - Webhooks is operating normally.
Mar 3, 05:20 UTC

Update - We are investigating intermittent connectivity issues between our backend and databases and will provide further updates as we have them. The current impact is you may see elevated latency while using our services.
Mar 3, 04:54 UTC

Update - We are seeing intermittent timeouts across our various services. We are currently investigating and will provide updates.
Mar 3, 04:23 UTC

Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Mar 3, 04:21 UTC

Investigating - We are investigating reports of degraded performance for API Requests, Git Operations and Issues
Mar 3, 04:20 UTC

Mar 2, 2025

No incidents reported.

Mar 1, 2025

Scheduled Codespaces Maintenance

Completed - The scheduled maintenance has been completed.
Mar 1, 02:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 28, 17:00 UTC

Scheduled - Codespaces will be undergoing maintenance in Europe and Southeast Asia from 17:00 UTC Friday February 28 to 02:00 UTC Saturday March 1. Maintenance will begin in North Europe at 17:00 UTC Friday February 28, followed by Southeast Asia, concluding in UK South. Each region will take 2-3 hours to complete.

During this time period, users may experience connectivity issues with new and existing Codespaces.

Please ensure that any uncommitted changes that you may need during the maintenance window are committed and pushed. Codespaces with any uncommitted changes will be accessible as usual once maintenance is complete.

Thank you for your patience as we work to improve our systems.
Feb 27, 21:09 UTC

Feb 28, 2025

Elevated Request Latency for Write operations on github.com and api.github.com

Resolved - This incident has been resolved.
Feb 28, 06:55 UTC

Update - Issues and Pull Requests are experiencing degraded performance. We are continuing to investigate.
Feb 28, 06:29 UTC

Investigating - We are currently investigating this issue.
Feb 28, 06:12 UTC

Feb 27, 2025

Disruption with some GitHub services

Resolved - This incident has been resolved.
Feb 27, 12:22 UTC

Update - The team is confident that recovery is complete. Thank you for your patience as this issue was investigated.
Feb 27, 12:22 UTC

Update - Our mitigations have rolled out successfully and have seen recovery for all Actions run starts back within expected range. Users should see Actions runs working normally.

We will keep this incident open for a short time while we continue to validate these results.
Feb 27, 12:16 UTC

Update - We have identified the cause of the delays to starting Action runs.

Our team is working to roll out mitigations and we hope to see recovery as these take effect in our systems over the next 10-20 minutes.

Further updates as we have more information.
Feb 27, 12:01 UTC

Update - We are seeing an increase in run start delays since 1104 UTC. This is impacting ~3% of Action runs at this time.

The team is working to understand the causes of this and to mitigate impact. We will continue to update as we have more information.
Feb 27, 11:39 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Feb 27, 11:31 UTC

Investigating - We are currently investigating this issue.
Feb 27, 11:28 UTC

Feb 26, 2025

Incident with Actions and Packages

Resolved - This incident has been resolved.
Feb 26, 17:19 UTC

Update - Actions and Packages are operating normally.
Feb 26, 17:19 UTC

Update - We're continuing our investigation into Billing interfaces and retrieval of packages causing Actions workflow run failures.
Feb 26, 16:41 UTC

Update - We’re investigating issues related to billing and the retrieval of packages that are causing Actions workflow run failures.
Feb 26, 16:17 UTC

Update - We're investigating issues related to the Billing interfaces and Packages downloads failing for enterprise customers.
Feb 26, 15:56 UTC

Investigating - We are investigating reports of degraded performance for Actions and Packages
Feb 26, 15:51 UTC

Feb 25, 2025

Disruption with some GitHub services

Resolved - On February 25th, 2025, between 14:25 UTC and 16:44 UTC email and web notifications experienced delivery delays. At the peak of the incident the delay resulted in ~10% of all notifications taking over 10 minutes to be delivered, with the remaining ~90% being delivered within 5-10 minutes. This was due to insufficient capacity in worker pools as a result of increased load during peak hours.

We also encountered delivery delays for a small number of webhooks, with delays of up-to 2.5 minutes to be delivered.

We mitigated the incident by scaling out the service to meet the demand.

The increase in capacity gives us extra headroom, and we are working to improve our capacity planning to prevent issues like this occurring in the future.
Feb 25, 16:50 UTC

Update - Web and email notifications are caught up, resolving the incident.
Feb 25, 16:49 UTC

Update - We're continuing to investigate delayed web and email notifications.
Feb 25, 16:16 UTC

Update - We're continuing to investigate delayed web and email notifications.
Feb 25, 15:43 UTC

Update - We're investigating delays in web and email notifications impacting all customers.
Feb 25, 15:13 UTC

Investigating - We are currently investigating this issue.
Feb 25, 15:12 UTC

Claude 3.7 Sonnet Partially Unavailable

Resolved - On February 25, 2025 between 13:40 UTC and 15:45 UTC the Claude 3.7 Sonnet model for GitHub Copilot Chat experienced degraded performance. During the impact, occasional requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.

We are working with our infrastructure providers to reduce time to detection and implement additional failover options, to mitigate issues like this one in the future.
Feb 25, 15:45 UTC

Update - We have disabled Claude 3.7 Sonnet models in Copilot Chat and across IDE integrations (VSCode, Visual Studio, JetBrains) due to an issue with our provider.

Users may still see these models as available for a brief period but we recommend switching to a different model. Other models were not impacted and are available.

Once our provider has resolved the issues impacting Claude 3.7 Sonnet models, we will re-enable them.
Feb 25, 15:25 UTC

Update - Copilot is experiencing degraded performance. We are continuing to investigate.
Feb 25, 14:44 UTC

Update - We are currently experiencing partial availability for the Claude 3.7 Sonnet and Claude 3.7 Thinking models in Copilot Chat, VSCode and other Copilot products. This is due to problems with an upstream provider. We are working to resolve these issues and will update with more information as it is made available.

Other Copilot models are available and working as expected.
Feb 25, 14:43 UTC

Investigating - We are currently investigating this issue.
Feb 25, 14:40 UTC

Incident with Packages

Resolved - On February 25, 2025, between 00:17 UTC and 01:08 UTC, GitHub Packages experienced a service degradation, leading to failures uploading and downloading packages, along with increased latency for all requests to GitHub Packages registry. At peak impact, about 14% of uploads and downloads failed, and all Packages requests were delayed by an average of 7 seconds. The incident was caused by the rollout of a database configuration change that resulted in a degradation in database performance. We mitigated the incident by rolling back the contributing change and failing over the database. In response to this incident, we are tuning database configurations and resolving a source of deadlocks. We are also redistributing certain workloads to read replicas to reduce latency and enhance overall database performance.
Feb 25, 01:08 UTC

Update - We have confirmed recovery for the majority of our systems. Some systems may still experience higher than normal latency as they catch up.
Feb 25, 01:08 UTC

Update - We have identified the issue impacting packages and have rolled out a fix. We are seeing signs of recovery and continue to monitor the situation.
Feb 25, 00:41 UTC

Investigating - We are investigating reports of degraded performance for Packages
Feb 25, 00:17 UTC

Feb 24, 2025

Claude 3.5 Sonnet model is unavailable in Copilot

Resolved - On February 24, 2025 between 21:42 UTC and 22:14 UTC the Claude 3.5 Sonnet model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude 3.5 Sonnet would result in an immediate error to the user. This was due to misconfiguration within one of our infrastructure providers that has since been mitigated.

We are working to prevent this error from occurring in the future by implementing additional failover options. Additionally we are updating our playbooks and alerting to reduce time to detection.
Feb 24, 22:14 UTC

Update - We were able to quickly identify the problem and resolve this issue. Claude 3.5 Sonnet is available again.
Feb 24, 22:14 UTC

Update - At this time, we are unable to serve requests to the Claude 3.5 Sonnet on Copilot. No other models are affected. We are investigating the issue and will provide updates as we discovery more information.
Feb 24, 22:08 UTC

Investigating - We are investigating reports of degraded performance for Copilot
Feb 24, 22:06 UTC

Disruption with some GitHub services

Resolved - On February 21 2025 12:00 UTC - 2/24/2025, 18:31 UTC, the Copilot Metrics API failed to ingest daily metrics aggregations for all customers resulting in failure to populate new metrics from 2025-02-21 to 2025-02-24. This failure was triggered by the metrics ingestion process timing out when querying across the event dataset. The API was functional for retrieving historical metrics prior to 2025-02-21.

On Monday morning 2/24/2025, 15:00 UTC, customer support was notified of the issue and the team deployed a fix to resolve query timeouts and ran backfills for the data from 2025-02-21 to 2025-02-23.

We are working to prevent further outages by adding more alerting to timeouts and have further optimized all our queries to aggregate data.
Feb 24, 18:31 UTC

Update - We have restored all of the data for 2025-02-21 to 2025-02-23. The data is queryable through the Copilot Metrics API. We are continuing to monitor the metrics data and expect to resolve the incident in the next hour.
Feb 24, 18:25 UTC

Update - We expect the missing data from the weekend to be available within two hours.
Feb 24, 17:50 UTC

Update - Copilot-metrics is in the process of restoring the usage statistics for 2025-02-23, we will continue to restore the previous 2 days over the next few hours.
Feb 24, 16:28 UTC

Update - Customers may not be able to review their usage statistics for copilot starting Saturday through Monday morning UTC. The API is functioning normally, but no data is available for those time periods. We are working on backfilling the data and all metrics will be eventually available later today. We estimate recovery within in the next few hours and will provide updates on this as the recovery process proceeds.
Feb 24, 15:56 UTC

Update - We are investigating reports of issues with service: Copilot metrics API. We will continue to keep users updated on progress towards mitigation.
Feb 24, 15:19 UTC

Investigating - We are currently investigating this issue.
Feb 24, 15:17 UTC

Incident with Issues

Resolved - This incident has been resolved.
Feb 24, 17:09 UTC

Update - Pull Requests is operating normally.
Feb 24, 17:09 UTC

Update - Issues is operating normally.
Feb 24, 17:09 UTC

Update - We continue to see recovery and expect Pull Requests and Issues search queries to recover within 30 minutes.
Feb 24, 16:48 UTC

Update - We are seeing recovery and expect for Pull Requests and Issues search queries to recover within 15 minutes.
Feb 24, 16:29 UTC

Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Feb 24, 16:14 UTC

Investigating - We are investigating reports of degraded performance for Issues
Feb 24, 16:08 UTC

Feb 23, 2025

No incidents reported.

Feb 22, 2025

No incidents reported.

Feb 21, 2025

No incidents reported.

Feb 20, 2025

No incidents reported.

Feb 19, 2025

No incidents reported.

Feb 18, 2025

No incidents reported.

Feb 17, 2025

No incidents reported.

About This Site