Incident with GIT LFS and Other Requests
Incident
Report for GitHub
Resolved
On February 6, 2025, between 8:40AM UTC and 11:13AM UTC the GitHub REST API was degraded following the rollout of a new feature. The feature resulted in an increase in requests that saturated a cache and led to cascading failures in unrelated services. The error rate peaked at 100% of requests to the service.
The incident was mitigated by increasing the allocated memory to the cache and rolling back the feature that led to the cache saturation. To prevent future incidents, we are working to reduce the time to detect a similar issue and optimize the overall calls to the cache.
The incident was mitigated by increasing the allocated memory to the cache and rolling back the feature that led to the cache saturation. To prevent future incidents, we are working to reduce the time to detect a similar issue and optimize the overall calls to the cache.
Update
This issue has been mitigated. We will continue to investigate root causes to ensure this does not reoccur.
Update
We have scaled out database resources and rolled back recent changes and are seeing signs of mitigation, but are monitoring to ensure complete recovery.
Update
We are attempting to scale databases to handle observed load spikes, as well as investigating other mitigation approaches.
Customers may intermittently experience failures to fetch repositories with LFS, as well as increased latency and errors across the API.
Customers may intermittently experience failures to fetch repositories with LFS, as well as increased latency and errors across the API.
Update
We are investigating failed Git LFS requests and potentially slow API requests.
Customers may experience failures to fetch repositories with LFS.
Customers may experience failures to fetch repositories with LFS.
Investigating
We are investigating reports of degraded performance for API Requests
This incident affected: API Requests.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe