Degraded snapshot upload latency

Timeline

1. 11:05 UTC Railway status page · Investigating We’re investigating elevated latency on build snapshot uploads in EU West (Amsterdam). Builds in this region may take longer than usual to start. Running deployments are not affected.
2. 11:25 UTC Railway status page · Reported Degraded snapshot upload latency
3. 11:30 UTC Railway status page · Identified We've identified the cause with our upstream storage provider. We're working with them to scale it up and restore normal upload times. Builds in EU West may still be slower than usual in the meantime.
4. 12:53 UTC Railway status page · Identified Due to sustained elevated latency, some builds in EU West may fail early. We’re still working with our provider to scale storage capacity.
5. 15:06 UTC Railway status page · Monitoring We’re seeing early signs of recovery in EU West. Snapshot upload times are improving and we’re monitoring closely.
6. 16:17 UTC Railway status page · Identified We're seeing elevated snapshot upload latency in US West (California) and Southeast Asia (Singapore), caused by the same issue with our upstream storage provider. Builds in these regions may take longer than usual to start. We're working with our provider to restore normal upload times. EU West continues to recover. Running deployments are not affected. Note: This is an issue with our upstream storage provider, unrelated to the security incident from October 21st, which has been resolved. No action is needed on your part.
7. 19:14 UTC Railway status page · Identified Ongoing
8. 20:29 UTC Railway status page · Identified We're still seeing elevated snapshot upload latency in US West (California), caused by the same issue with our upstream storage provider. Builds in this region may take longer than usual to start. We're working with our storage provider to restore normal upload times. EU West and Southeast Asia are now in recovery. Running deployments are not affected. Note: This is an issue with our upstream storage provider, unrelated to the security incident from October 21st, which has been resolved. No action is needed on your part.
9. 22:57 UTC Railway status page · Monitoring This latency is now normalizing in US West also, all other regions should be fully back to normal. We continue to monitor.
10. 23:43 UTC Railway status page · Resolved Snapshot upload latency has now resolved across all regions.