tag:status.sentry.io,2005:/historySentry Status - Incident History2024-03-19T01:48:46-07:00Sentrytag:status.sentry.io,2005:Incident/202727712024-03-15T12:05:00-07:002024-03-15T13:38:25-07:00Some events dropped from Asia and Europe traffic<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:05</var> PDT</small><br><strong>Resolved</strong> - From 19:05 to 19:15UTC some events (fewer than 1%) sent from Europe and Asia region clients were dropped by our ingestion pipeline. The configuration change which caused the incident has been identified and resolved.</p>tag:status.sentry.io,2005:Incident/201085952024-02-29T10:53:47-08:002024-02-29T10:53:47-08:00Errors Ingestion Issue<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>10:53</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>10:05</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for further issues.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:44</var> PST</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring the results.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:21</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue, and will keep providing updates as we recover.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>08:34</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. We will continue to provide further updates.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>08:27</var> PST</small><br><strong>Update</strong> - We are continuing to investigate the issue and will provide further updates.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>07:56</var> PST</small><br><strong>Investigating</strong> - We're currently investigating reports of a potential issue with error ingestion and will provide further updates as soon as we have more information.</p>tag:status.sentry.io,2005:Incident/201071832024-02-29T06:38:20-08:002024-02-29T06:38:20-08:00Attachment delays<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>06:38</var> PST</small><br><strong>Resolved</strong> - Between 7:30 - 9:30am EST, we experienced a delay in attachment ingestion. This was caused by increased latency on one of our storage systems. The issue was resolved and attachments ingestion is operating as expected.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>06:24</var> PST</small><br><strong>Monitoring</strong> - Attachment ingestion latency is below 1 minute. We're continuing to monitor for any regression.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>05:59</var> PST</small><br><strong>Update</strong> - We're continuing to investigate this issue. Attachments were delayed by up to 50 minutes. New attachments are currently delayed by about 30 minutes and we're continuing to catch up on delayed attachments.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>04:30</var> PST</small><br><strong>Investigating</strong> - We're currently investigating reports of an issue causing delays in attachments ingestion and will provide further updates as soon as we have more information.</p>tag:status.sentry.io,2005:Incident/200909772024-02-28T11:47:50-08:002024-02-28T11:47:51-08:00Delivery of Slack notifications is failing for a small number of organizations.<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>11:47</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Investigating</strong> - Delivery of Slack notifications is failing for a small number of organizations. Investigations are ongoing.</p>tag:status.sentry.io,2005:Incident/200830152024-02-26T18:16:32-08:002024-02-26T18:16:32-08:00Sentry API outage<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>18:16</var> PST</small><br><strong>Resolved</strong> - This incident has been successfully resolved.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>15:13</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>15:13</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>15:12</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.sentry.io,2005:Incident/200624912024-02-23T18:42:44-08:002024-02-23T18:42:45-08:00Crons Monitors ingestion delayed<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>18:42</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>18:25</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>18:09</var> PST</small><br><strong>Investigating</strong> - We are currently looking into a delay in ingesting crons monitors.</p>tag:status.sentry.io,2005:Incident/200207232024-02-19T06:03:15-08:002024-02-19T06:03:15-08:00Delays in span processing<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>06:03</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>05:47</var> PST</small><br><strong>Identified</strong> - Starting at 5:15 AM PST, due to an issue in the span processing pipeline, we experienced delays in span processing of up to 5 minutes. The issue has been identified and fixed.</p>tag:status.sentry.io,2005:Incident/200025292024-02-16T13:04:05-08:002024-02-16T13:04:06-08:00Event processing delay<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>13:04</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>12:50</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>12:38</var> PST</small><br><strong>Identified</strong> - We are currently experiencing an event processing delay due to the failure of one of our key-value stores.</p>tag:status.sentry.io,2005:Incident/199793232024-02-13T15:36:36-08:002024-02-13T15:36:36-08:00Delays in Errors processing<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>15:36</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>15:08</var> PST</small><br><strong>Monitoring</strong> - We've implemented a fix for our delayed errors ingestion and are monitoring the results.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>15:00</var> PST</small><br><strong>Identified</strong> - We've identified an issue causing delays in our errors ingest pipeline and are working on a fix.</p>tag:status.sentry.io,2005:Incident/199693882024-02-12T13:08:02-08:002024-02-12T13:08:02-08:00sentry.io returns 502<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:08</var> PST</small><br><strong>Resolved</strong> - A primary database failover caused a brief outage of the site, but no data was lost. The reverted deploy was coincidental to the failover, but it was not a contributing factor.<br /><br />All services should be functioning normally now.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>12:15</var> PST</small><br><strong>Monitoring</strong> - We've reverted a recent deploy, and are monitoring the effects. Dashboards should be available again.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>12:14</var> PST</small><br><strong>Investigating</strong> - We're currently investigating an issue on our dashboards.</p>tag:status.sentry.io,2005:Incident/199483322024-02-09T12:44:17-08:002024-02-09T12:44:17-08:00Delayed transaction ingestion<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>12:44</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>12:43</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>08:46</var> PST</small><br><strong>Monitoring</strong> - Our transaction ingestion delay has been fixed, we're continuing to monitor.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>07:31</var> PST</small><br><strong>Update</strong> - We're continuing to investigate a delay in our transaction ingestion.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>06:41</var> PST</small><br><strong>Investigating</strong> - We're investigating a delay in transaction ingestion.</p>tag:status.sentry.io,2005:Incident/199365772024-02-08T13:19:32-08:002024-02-08T13:19:32-08:00Delays in processing, failures with file operations<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>13:19</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>11:49</var> PST</small><br><strong>Monitoring</strong> - The transactions ingestion delay has been fixed. We're continuing to monitor.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>11:14</var> PST</small><br><strong>Identified</strong> - Ingestion of transactions is currently delayed and we're continuing to investigate.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>10:51</var> PST</small><br><strong>Investigating</strong> - Ingestion of errors and transactions is currently delayed and we're continuing to investigate.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>07:02</var> PST</small><br><strong>Update</strong> - We are continuing to process the historical data; updated estimate: the remaining data will be processed in ~90 minutes.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>04:05</var> PST</small><br><strong>Update</strong> - We have resumed ingesting new events and are actively working on backfilling older events. The backfill process is estimated to be completed within the next 3 hours.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>02:43</var> PST</small><br><strong>Monitoring</strong> - We're continuing to work on this issue. We've implemented another set of fixes and monitoring the results.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>01:36</var> PST</small><br><strong>Identified</strong> - We're investigating a related issue, source map uploads might be failing at the moment.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>00:51</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>00:16</var> PST</small><br><strong>Monitoring</strong> - We've applied a fix, and currently monitoring the results. The new events should be ingested and processed properly now, while the older events are being backfilled.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>23:31</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>22:57</var> PST</small><br><strong>Update</strong> - We are continuing to investigate the issue with failures around file-based API and operations.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>21:57</var> PST</small><br><strong>Investigating</strong> - We’re investigating delays in processing and failures around file-related operations.</p>tag:status.sentry.io,2005:Incident/199317702024-02-07T08:11:17-08:002024-02-07T08:11:17-08:00Authentication and UI errors<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>08:11</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>08:10</var> PST</small><br><strong>Monitoring</strong> - Between 15:22 and 15:53 UTC we experienced authentication issues and other UI slowdowns, the root cause has been identified and the issue has been resolved.</p>tag:status.sentry.io,2005:Incident/199200662024-02-06T02:39:23-08:002024-02-06T02:39:23-08:00Delays with Session Replay Processing<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>02:39</var> PST</small><br><strong>Resolved</strong> - The incident has been resolved, Session Replay ingestion and processing is fully operational again.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>02:13</var> PST</small><br><strong>Monitoring</strong> - We've applied a fix and monitoring the results.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>01:33</var> PST</small><br><strong>Investigating</strong> - We are investigating issues with session replays; ingestion and processing of session replay is currently delayed.</p>tag:status.sentry.io,2005:Incident/199202982024-02-06T02:31:16-08:002024-02-06T02:31:16-08:00Issues with JavaScript Loader<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>02:31</var> PST</small><br><strong>Resolved</strong> - We've applied a fix, and the issue should be resolved now.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>02:18</var> PST</small><br><strong>Investigating</strong> - We’re investigating an availability issue with JavaScript loader script (HTTP 502 responses)</p>tag:status.sentry.io,2005:Incident/199136582024-02-05T09:18:41-08:002024-02-05T09:18:41-08:00UI and errors ingestion problem<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>09:18</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>08:13</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>08:13</var> PST</small><br><strong>Identified</strong> - We identified an issue with our UI page returning results and errors ingestion that we have mitigated. Our UI would have returned only cached results and errors ingestion would have been interrupted between 15:41 UTC and 16:04 UTC.</p>tag:status.sentry.io,2005:Incident/198865772024-02-01T10:36:49-08:002024-02-01T10:36:49-08:00Event ingestion Delayed<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>10:36</var> PST</small><br><strong>Resolved</strong> - The incident is now mitigated.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>09:35</var> PST</small><br><strong>Investigating</strong> - Started from 16:45 UTC, we experienced an issue in ingestion and event processing for all event types that caused delays</p>tag:status.sentry.io,2005:Incident/198232192024-01-24T16:49:06-08:002024-01-24T16:49:07-08:00JavaScript CDN issues<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>16:49</var> PST</small><br><strong>Resolved</strong> - The rate of 404s against our JavaScript CDN has returned to normal after a configuration change. Closing this incident as resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>15:12</var> PST</small><br><strong>Investigating</strong> - We are investigating an issue causing our JavaScript CDN to occasionally throw 404s.</p>tag:status.sentry.io,2005:Incident/199945472024-01-21T16:00:00-08:002024-02-15T12:17:18-08:00Intermittent chunk-upload failures Jan 21-26<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>16:00</var> PST</small><br><strong>Resolved</strong> - Between January 21st and January 26th our chunk-upload endpoint experienced intermittent HTTP 50x errors due to an internal buffer issue which was resolved on the 26th.</p>tag:status.sentry.io,2005:Incident/197461662024-01-17T21:33:23-08:002024-01-17T21:33:25-08:00Issue page not loading and Event Ingestion delays<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>21:33</var> PST</small><br><strong>Resolved</strong> - The issues causing event ingestion delays have been resolved.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>19:10</var> PST</small><br><strong>Monitoring</strong> - We have implemented a fix for our event ingestion delays and are monitoring the results.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>18:52</var> PST</small><br><strong>Identified</strong> - We have identified the previous issues. The Issues page has been fixed, and we're working on a fix for our event ingestion delays.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>18:20</var> PST</small><br><strong>Investigating</strong> - We are investigating issues with our Issue Page, as well as Error and Transaction ingestion.</p>tag:status.sentry.io,2005:Incident/197164382024-01-15T13:39:05-08:002024-01-15T13:39:06-08:00Spans Ingestion Errors<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>13:39</var> PST</small><br><strong>Resolved</strong> - Spans ingestion is functioning normally again.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>13:36</var> PST</small><br><strong>Monitoring</strong> - The fix has been deployed and we are monitoring the results.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>13:27</var> PST</small><br><strong>Identified</strong> - A problem has been identified that resulted in errors during spans ingestion in a small percentage of cases. A fix is being deployed.</p>tag:status.sentry.io,2005:Incident/196898102024-01-12T15:16:33-08:002024-01-12T15:16:33-08:00Replay Quotas reset for some customers<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:16</var> PST</small><br><strong>Resolved</strong> - The issue with replay quotas has been resolved.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>13:56</var> PST</small><br><strong>Update</strong> - A fix has been deployed, and we're monitoring the results.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>13:55</var> PST</small><br><strong>Monitoring</strong> - A fix has been deployed, and we're monitoring the results.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>11:17</var> PST</small><br><strong>Update</strong> - We've identified the issue causing replay quotas to be reset for some customers. We have mitigated the issue and are in the process of deploying a permanent fix.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>10:14</var> PST</small><br><strong>Identified</strong> - We've identified the issue causing replay quotas to be reset for some customers and are working on a fix.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>09:38</var> PST</small><br><strong>Update</strong> - We are continuing to investigate the issue with replay billing quotas, and will post updates as they come.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>08:42</var> PST</small><br><strong>Investigating</strong> - We are investigating an issue with replay quotas being reset for some customers.</p>tag:status.sentry.io,2005:Incident/196809162024-01-11T11:45:04-08:002024-01-11T11:45:05-08:00discover/profiling homepage events not displaying.<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:45</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:32</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:25</var> PST</small><br><strong>Investigating</strong> - We’ve identified an issue where the discover and profiling homepages do not render event counts correctly. We are actively investigating the issue</p>tag:status.sentry.io,2005:Incident/196134032024-01-04T10:56:13-08:002024-01-04T10:56:13-08:00Ingestion Delays<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>10:56</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>09:17</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results. There was some data loss in transaction metrics and release health between 16:10 UTC and 17:04 UTC.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>08:57</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>08:57</var> PST</small><br><strong>Investigating</strong> - We are investigating an issue causing ingestion delays in several of our products.</p>tag:status.sentry.io,2005:Incident/196036632024-01-03T09:41:48-08:002024-01-03T09:41:48-08:00Delayed Profile Ingestion<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>09:41</var> PST</small><br><strong>Resolved</strong> - The issue with Profiling ingestion has been resolved.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Monitoring</strong> - We've implemented a fix and are monitoring the results.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>08:46</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue causing delayed ingestion for Profiling.</p>