Resolved -
This incident has been resolved.
Oct 1, 20:15 PDT
Monitoring -
The transaction backlog has finished processing. All ingestion services are operating normally again.
Oct 1, 19:30 PDT
Update -
We are still processing transactions produced prior to 15:30 PM UTC and aim to be completed within half an hour.
Oct 1, 18:58 PDT
Update -
We are still processing transactions produced prior to 15:30 PM UTC. The processing has taken longer than anticipated. We estimate another hour until they're all processed.
Oct 1, 17:24 PDT
Update -
We are still processing transactions produced prior to 15:30 PM UTC. We estimate another hour and a half until they're all processed.
Oct 1, 15:53 PDT
Update -
Transactions produced prior to 15:30 PM UTC are still in the backlog, we estimate another 7 hours from now until they’re all processed.
Oct 1, 12:51 PDT
Update -
We’re currently ingesting all events in real-time (including errors and transaction).
All errors produced prior to this outage have been processed. All transactions produced prior to 15:30 PM UTC are still in the backlog. We will provide an ETA for when that will be completed when the burn rate stabilizes.
Oct 1, 11:24 PDT
Update -
We're continuing the parallel ingest current events and backfill older events.
Oct 1, 10:04 PDT
Update -
We've implemented the change to prioritize the ingestion of more recent events and continue to process our backlog to backfill older events.
Oct 1, 08:45 PDT
Update -
We're continuing to process our event ingestion backlog in our US region and are working on a change to prioritize the ingestion of recent events.
Oct 1, 08:04 PDT
Update -
Starting at 8:10 am UTC we encountered an issue which required us to restart some ingestion services. During this activity we encountered a defect in the underlying platform which triggered an ingestion backlog for all US event types. We've worked with our cloud provider to implement a mitigation until they can provide a long term fix. We're currently processing our event ingestion backlog and continuing to monitor.
Oct 1, 07:21 PDT
Update -
We're continuing to investigate a fix for our ingestion backlog in our US region, ingestion of all event types is delayed in this region.
Oct 1, 04:52 PDT
Identified -
We've identified a problem causing excessive backlogging, and are working on a fix for it now.
Oct 1, 03:47 PDT
Investigating -
We're currently investigating ingestion delays in the US region.
Oct 1, 02:28 PDT