Delays in processing, failures with file operations

Incident Report for Sentry

Resolved

This incident has been resolved.
Posted Feb 08, 2024 - 21:19 UTC

Monitoring

The transactions ingestion delay has been fixed. We're continuing to monitor.
Posted Feb 08, 2024 - 19:49 UTC

Identified

Ingestion of transactions is currently delayed and we're continuing to investigate.
Posted Feb 08, 2024 - 19:14 UTC

Investigating

Ingestion of errors and transactions is currently delayed and we're continuing to investigate.
Posted Feb 08, 2024 - 18:51 UTC

Update

We are continuing to process the historical data; updated estimate: the remaining data will be processed in ~90 minutes.
Posted Feb 08, 2024 - 15:02 UTC

Update

We have resumed ingesting new events and are actively working on backfilling older events. The backfill process is estimated to be completed within the next 3 hours.
Posted Feb 08, 2024 - 12:05 UTC

Monitoring

We're continuing to work on this issue. We've implemented another set of fixes and monitoring the results.
Posted Feb 08, 2024 - 10:43 UTC

Identified

We're investigating a related issue, source map uploads might be failing at the moment.
Posted Feb 08, 2024 - 09:36 UTC

Update

We are continuing to monitor for any further issues.
Posted Feb 08, 2024 - 08:51 UTC

Monitoring

We've applied a fix, and currently monitoring the results. The new events should be ingested and processed properly now, while the older events are being backfilled.
Posted Feb 08, 2024 - 08:16 UTC

Update

We are continuing to investigate this issue.
Posted Feb 08, 2024 - 07:31 UTC

Update

We are continuing to investigate the issue with failures around file-based API and operations.
Posted Feb 08, 2024 - 06:57 UTC

Investigating

We’re investigating delays in processing and failures around file-related operations.
Posted Feb 08, 2024 - 05:57 UTC
This incident affected: US Ingestion (US Error Ingestion, US Transaction Ingestion, US Attachment Ingestion) and API.