The transaction backlog has finished processing. All ingestion services are operating normally again.
Posted Oct 01, 2024 - 19:30 PDT
Update
We are still processing transactions produced prior to 15:30 PM UTC and aim to be completed within half an hour.
Posted Oct 01, 2024 - 18:58 PDT
Update
We are still processing transactions produced prior to 15:30 PM UTC. The processing has taken longer than anticipated. We estimate another hour until they're all processed.
Posted Oct 01, 2024 - 17:24 PDT
Update
We are still processing transactions produced prior to 15:30 PM UTC. We estimate another hour and a half until they're all processed.
Posted Oct 01, 2024 - 15:53 PDT
Update
Transactions produced prior to 15:30 PM UTC are still in the backlog, we estimate another 7 hours from now until they’re all processed.
Posted Oct 01, 2024 - 12:51 PDT
Update
We’re currently ingesting all events in real-time (including errors and transaction). All errors produced prior to this outage have been processed. All transactions produced prior to 15:30 PM UTC are still in the backlog. We will provide an ETA for when that will be completed when the burn rate stabilizes.
Posted Oct 01, 2024 - 11:24 PDT
Update
We're continuing the parallel ingest current events and backfill older events.
Posted Oct 01, 2024 - 10:04 PDT
Update
We've implemented the change to prioritize the ingestion of more recent events and continue to process our backlog to backfill older events.
Posted Oct 01, 2024 - 08:45 PDT
Update
We're continuing to process our event ingestion backlog in our US region and are working on a change to prioritize the ingestion of recent events.
Posted Oct 01, 2024 - 08:04 PDT
Update
Starting at 8:10 am UTC we encountered an issue which required us to restart some ingestion services. During this activity we encountered a defect in the underlying platform which triggered an ingestion backlog for all US event types. We've worked with our cloud provider to implement a mitigation until they can provide a long term fix. We're currently processing our event ingestion backlog and continuing to monitor.
Posted Oct 01, 2024 - 07:21 PDT
Update
We're continuing to investigate a fix for our ingestion backlog in our US region, ingestion of all event types is delayed in this region.
Posted Oct 01, 2024 - 04:52 PDT
Identified
We've identified a problem causing excessive backlogging, and are working on a fix for it now.
Posted Oct 01, 2024 - 03:47 PDT
Investigating
We're currently investigating ingestion delays in the US region.
Posted Oct 01, 2024 - 02:28 PDT
This incident affected: US Ingestion (US Error Ingestion, US Transaction Ingestion, US Attachment Ingestion, US Profile Ingestion, US Replay Ingestion, US Span Ingestion, US Cron Monitoring, Custom Metrics (beta)).