Resolved -
Event processing is now functioning normally.
Jun 12, 21:19 UTC
Update -
Upstream is slowly recovering and our ingestion rate has increased. We will continue to monitor the situation as we move toward backlog processing.
Jun 12, 20:39 UTC
Update -
We continue to monitor the upstream incident as we process incoming events as quickly as possible.
Jun 12, 20:15 UTC
Identified -
Google Cloud Platform has begin to mitigate the problem (https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Though slowly, we continue to ingest and process events during this incident and we will be able to begin the process of burning down backlogs after recovery.
Jun 12, 19:25 UTC
Update -
We have received acknowledgement from our cloud provider that there is a wide-scale impact hitting their services. We will continue to work with them and prepare our systems for the return of service.
Jun 12, 18:46 UTC
Update -
We have identified the problem as being with our cloud infrastructure provider and have contacted them for further diagnosis. Customer impact is delays in ingestion and intermittent failures for both EU and US customers.
Jun 12, 18:29 UTC
Investigating -
We are currently investigating this issue which is impacting both US and EU localities.
Jun 12, 18:10 UTC