Affected
- ResolvedResolved
Snowflake has coordinated with their third-party service provider to implement the fix for this issue, and they have monitored the environment to confirm that service was restored.
Incident start time: 07:22 UTC July 05, 2023
Incident end time: 16:00 UTC July 05, 2023 - MonitoringUpdate
Snowflake continue to make progress on recovery, and packet drops in the region have now returned to pre-incident levels. We expect that our onsite repairs and augmentations will be able to bring sufficient additional capacity online in the next 2-3 hours, which will provide headroom before the Thursday business day starts in EMEA. The next update will be provided in 2 hours, or as events warrant.
This message was last updated at 21:12 UTC on 05 July 2023
- MonitoringMonitoring
MS Azure's last update
Efforts to repair existing capacity are making good progress - crews from our fiber partners have been able to access the area, and have partially restored the impacted links. We are also accelerating efforts to augment networking capacity, with onsite teams preparing relevant cabling to bring additional capacity between datacenters online. We expect that our onsite repairs and augmentations will be able to bring sufficient additional capacity online in the next few hours, which will provide headroom before the Thursday business day starts in EMEA.
This message was last updated at 19:29 UTC on 05 July 2023
- IdentifiedIdentified
Latest MS Azure update:
Packet drops have reduced significantly, so the vast majority of services in the region are functioning as expected. Traffic on a few specific networking links are still experiencing higher than usual latency. We continue to work on restoring the networking capacity that was lost by the fiber cut(s) and, in parallel, we are accelerating efforts to bring additional net new capacity online. Efforts to repair existing capacity are currently being hindered by the ongoing weather situation in the Netherlands, as there are areas where it is not yet safe for people to work.We are monitoring the network infrastructure and capacity, and will declare this incident as mitigated once we are confident we have sufficient capacity to avoid future availability or performance issues. The next update will be provided in 2 hours, or as events warrant.
- MonitoringMonitoring
Here is MS Azure's last update:
Starting at 07:22 UTC on 05 Jul 2023, a subset of Azure customers in West Europe may experience packet drops, timeouts and increased latencies in responses. Automated systems are balancing load and rerouting traffic, but customers may still experience degradation.
We recommend that customers that are experiencing impact invoke their BCDR plans and fail over to another region if possible.
Current Status: We determined a number of links between two datacenters have become unavailable due to a fiber cut, caused by severe weather conditions in the Netherlands, predominantly impacting communications between these datacenters in the West Europe region. As a result of this interruption, Azure services that rely on internal communications with other services may experience degraded performance.
We are continuing to apply mitigation actions and monitoring the status of this issue, we have been seeing signs of recovery in the last 30 minutes. The next update will be provided in 2 hours, or as events warrant. - IdentifiedIdentified
We would like to provide you with an important update regarding the ongoing service disruption. Snowflake, in close collaboration with their third-party provider, is actively working to restore the affected services. We are pleased to inform you that they are anticipating a resolution within the next few hours.
- InvestigatingInvestigating
We wish to inform you about a situation that has come to our attention regarding Snowflake's services. Specifically, they have identified an issue with one of their third-party service providers, and unfortunately, it is currently impacting the functionality of some of our products.
Please rest assured that our dedicated team is actively monitoring the situation closely. We are committed to providing you with regular updates as soon as we have new information to share.