Between 10:30 and 12:15 pacific time, we experienced ingestion delays and dropped message in the errors, transactions and attachments ingestion pipelines, due to a component failure related to underlying infrastructure from our cloud provider. Roughly 1 in 1000 messages were not processed, and dropped events did not count against customer quotas. The issue is now resolved.
Posted Dec 23, 2024 - 14:13 PST
Update
Ingest latencies have recovered, and we are no longer dropping events. We estimate that 1 in 1000 events were dropped between 10:30 and 12:00 Pacific Time. The maximum ingestion delay was around 35 minutes. Dropped events were not counted against customer quotas.
Posted Dec 23, 2024 - 12:20 PST
Update
We are continuing to monitor for any further issues.
Posted Dec 23, 2024 - 12:18 PST
Monitoring
Ingest latencies have recovered, and we are no longer dropping events. We estimate that 1 in 1000 events were dropped between 10:30 and 12:00 Pacific Time. The maximum ingestion delay was around 35 minutes. Dropped events were not counted against customer quotas.
Posted Dec 23, 2024 - 12:17 PST
Investigating
We are currently experiencing delays and dropped messages on the events and transactions ingestion pipelines in the US region. We are currently working with our cloud provider to investigate the issue.
Posted Dec 23, 2024 - 11:08 PST
This incident affected: US Ingestion (US Error Ingestion, US Transaction Ingestion, US Attachment Ingestion).