Resolved -
We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types.
Jul 20, 16:57 UTC
Update -
We are still burning the backlog of EU region error events received during the incident. We are around halfway through the backlog.
Jul 20, 16:09 UTC
Update -
The backlog on attachments in our EU region is fully burned, and latency is back to normal. While new error events are being processed with normal latency, we have a backlog of old error events that we are now burning through.
Jul 20, 14:50 UTC
Monitoring -
We have implemented a fix for our attachment ingestion issue in our EU region, and are monitoring while the backlog burns. We will update here when latency is back to normal.
Jul 20, 14:30 UTC
Update -
We have implemented a fix for errors and feedback ingestion in our EU region, and latency for those pipelines has returned to normal. We are still working on fixing our attachments ingestion pipeline in EU.
Jul 20, 13:47 UTC
Update -
We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.
Jul 20, 13:19 UTC
Update -
We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.
Jul 20, 12:24 UTC
Update -
We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.
Jul 20, 12:10 UTC
Identified -
We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are working on a fix.
Jul 20, 11:41 UTC
Investigating -
we are investigating an issue affecting errors ingestion in the DE/EU region
Jul 20, 11:11 UTC