Delay in data appearing in UI and API

Timeline

1. 11:33 UTC CircleCI status page · Reported Delay in data appearing in UI and API
2. 19:29 UTC CircleCI status page · Identified Since our last update, jobs are starting faster: most are now starting within about 6 minutes, down from about 8 minutes. The CircleCI UI is catching up but still about an hour behind. New pipelines are being processed as they arrive. Jobs may already have run even if the UI does not show them yet, so please avoid retriggering workflows that don't appear in the UI. We will provide another update by 20:00 UTC.
3. 20:06 UTC CircleCI status page · Identified Jobs are starting faster and fewer are failing: most jobs are now starting within about 4 minutes, down from about 6 minutes, and the longest waits have dropped from about 30 minutes to about 20 minutes. Pipelines from earlier in the incident are now being worked through. The CircleCI UI is still behind, and some older updates may take longer to appear. Jobs may already have run even if the UI does not show them yet, so please avoid retriggering workflows that don't appear in the UI. We will provide another update by 20:30 UTC.
4. 21:22 UTC CircleCI status page · Identified Since about 20:48 UTC, new workflows and jobs are not starting. We have temporarily paused starting new work so that our systems can catch up on work already in progress, including delayed updates to the CircleCI UI and API. Older updates are now appearing in the UI and API as this backlog clears. New pipelines are still being accepted and queued Please avoid retriggering pipelines or workflows during this time, as this may create duplicates once work resumes. We will provide another update by 22:00 UTC.
5. 22:09 UTC CircleCI status page · Identified Since our last update, most of the pipelines that were delayed earlier in the incident have now been processed, and their workflows are queued to start. New workflows and jobs remain paused while we prepare to resume work safely. Delayed updates continue to appear in the CircleCI UI and API. Please continue to avoid retriggering pipelines or workflows, as this may create duplicates once work resumes. We will provide another update by 22:30 UTC.
6. 22:28 UTC CircleCI status page · Identified Since our last update, we have begun gradually resuming work, and some queued workflows and jobs are starting again. Most queued work is still waiting, and jobs that do start may have waited over an hour. We are increasing the rate carefully to avoid overloading our systems. Delayed updates continue to appear in the CircleCI UI and API. Please continue to avoid retriggering pipelines or workflows, as this may create duplicates as queued work resumes. We will provide another update by 23:00 UTC.
7. 23:00 UTC CircleCI status page · Identified A large number of queued workflows and jobs have started, and the CircleCI UI and API have caught up and are showing updates normally with few delays. We are continuing to work through the remaining queued work. Jobs from the queue may have waited about an hour or more to start, and some Linux jobs are waiting up to about 10 minutes for capacity. Please continue to avoid retriggering while queued work continues to process. We will provide another update by 23:30 UTC.
8. 23:38 UTC CircleCI status page · Identified We are still working through a significant backlog of queued workflows and jobs. Wait times are improving but they remain longer than normal and some jobs have waited over an hour. The CircleCI UI and API remain up to date. Please avoid retriggering pipelines or workflows while the backlog clears, as this may create duplicates. We will provide another update by 00:00 UTC.
9. 23:59 UTC CircleCI status page · Identified Since our last update, the backlog of queued workflows and jobs has cleared, as of about 23:47 UTC. Workflows and jobs are now starting within seconds, and the CircleCI UI and API are up to date. We are monitoring our systems closely to make sure they remain stable. We will provide another update by 00:30 UTC.
10. 00:45 UTC CircleCI status page · Identified Workflows and jobs have continued to start normally, within seconds, and the CircleCI UI and API remain up to date. Slack and email notifications delayed earlier in the incident are still being delivered. Usage charge data on the Project Usage page is stale. We are continuing to monitor our systems closely and will provide another update by 01:15 UTC.
11. 01:23 UTC CircleCI status page · Monitoring Workflows and jobs are starting and running normally, and the CircleCI UI and API are up to date. We are continuing to monitor our systems closely to make sure they remain stable. We will provide another update by 02:00 UTC.
12. 02:08 UTC CircleCI status page · Resolved Between 11:15 UTC and 23:47 UTC on October 9, customers experienced delays and failures across pipelines, workflows and jobs, along with delays in the CircleCI UI, API and notifications. From 11:15 to about 13:00 UTC, pipelines were created but not processed, and outbound webhooks and Slack and email notifications were not sent. From about 13:00 to 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. From about 15:15 to 20:48 UTC, workflows and jobs started with increasing delays, of up to about an hour, and the CircleCI UI and API fell behind by up to about…
13. 22:32 UTC CircleCI status page · Postmortem ## Summary Between 11:15 UTC and 23:47 UTC on October 9, CircleCI customers experienced hours-long delays and failures across pipelines, workflows and jobs, along with delays in the CircleCI UI, API and notifications. * From 11:15 to 13:00 UTC, pipelines were created but not processed, and outbound webhooks and Slack and email notifications were not sent. * From 13:00 to 14:40 UTC, workflows and jobs started with delays while we worked through the backlog. * From 15:15 to 20:48 UTC, workflows and jobs started with increasing delays, of up to about an hour, and the CircleCI UI and API fell…