Between 10:30 and 12:15 pacific time, we experienced ingestion delays and dropped message in the errors, transactions and attachments ingestion pipelines, due to a component failure related to underlying infrastructure from our cloud provider. Roughly 1 in 1000 messages were not processed, and dropped events did not count against customer quotas. The issue is now resolved.
On November 27 between 21:46 - 21:57 UTC, there was a denial of service attack on Sentry sending up to 10x the normal volume of traffic. Our systems scaled up within 6 minutes to accommodate this traffic. Between 21:46 - 21:52 UTC some customers may have experienced traffic loss of up to 80%.
We are implementing changes to our ingestion layer to reduce the impact of similar attacks in the future.