Snowflake issue affecting service

Resolved
Degraded performance
Started 11 months ago Lasted about 13 hours

Affected

Flight Info Direct
Flight Info API
Flight Info Alerts
Updates
  • Resolved
    Resolved

    Snowflake has coordinated with their third-party service provider to implement the fix for this issue, and they have monitored the environment to confirm that service was restored.

    Incident start time: 07:22 UTC July 05, 2023
    Incident end time: 16:00 UTC July 05, 2023

  • Monitoring
    Update

    Snowflake continue to make progress on recovery, and packet drops in the region have now returned to pre-incident levels. We expect that our onsite repairs and augmentations will be able to bring sufficient additional capacity online in the next 2-3 hours, which will provide headroom before the Thursday business day starts in EMEA. The next update will be provided in 2 hours, or as events warrant.

    This message was last updated at 21:12 UTC on 05 July 2023

  • Monitoring
    Monitoring

    MS Azure's last update

    Efforts to repair existing capacity are making good progress - crews from our fiber partners have been able to access the area, and have partially restored the impacted links. We are also accelerating efforts to augment networking capacity, with onsite teams preparing relevant cabling to bring additional capacity between datacenters online. We expect that our onsite repairs and augmentations will be able to bring sufficient additional capacity online in the next few hours, which will provide headroom before the Thursday business day starts in EMEA.

    This message was last updated at 19:29 UTC on 05 July 2023

  • Identified
    Identified

    Latest MS Azure update:
    Packet drops have reduced significantly, so the vast majority of services in the region are functioning as expected. Traffic on a few specific networking links are still experiencing higher than usual latency. We continue to work on restoring the networking capacity that was lost by the fiber cut(s) and, in parallel, we are accelerating efforts to bring additional net new capacity online. Efforts to repair existing capacity are currently being hindered by the ongoing weather situation in the Netherlands, as there are areas where it is not yet safe for people to work.

    We are monitoring the network infrastructure and capacity, and will declare this incident as mitigated once we are confident we have sufficient capacity to avoid future availability or performance issues. The next update will be provided in 2 hours, or as events warrant.

  • Monitoring
    Monitoring

    Here is MS Azure's last update:

    Starting at 07:22 UTC on 05 Jul 2023, a subset of Azure customers in West Europe may experience packet drops, timeouts and increased latencies in responses. Automated systems are balancing load and rerouting traffic, but customers may still experience degradation.
    We recommend that customers that are experiencing impact invoke their BCDR plans and fail over to another region if possible.
    Current Status: We determined a number of links between two datacenters have become unavailable due to a fiber cut, caused by severe weather conditions in the Netherlands, predominantly impacting communications between these datacenters in the West Europe region. As a result of this interruption, Azure services that rely on internal communications with other services may experience degraded performance.
    We are continuing to apply mitigation actions and monitoring the status of this issue, we have been seeing signs of recovery in the last 30 minutes. The next update will be provided in 2 hours, or as events warrant.

  • Identified
    Identified

    We would like to provide you with an important update regarding the ongoing service disruption. Snowflake, in close collaboration with their third-party provider, is actively working to restore the affected services. We are pleased to inform you that they are anticipating a resolution within the next few hours.

  • Investigating
    Investigating

    We wish to inform you about a situation that has come to our attention regarding Snowflake's services. Specifically, they have identified an issue with one of their third-party service providers, and unfortunately, it is currently impacting the functionality of some of our products.

    Please rest assured that our dedicated team is actively monitoring the situation closely. We are committed to providing you with regular updates as soon as we have new information to share.