OAG - Notice history

The issue has been addressed and resolved. We are currently monitoring the result.

The production and delivery of all the WDF and SSIM files have now been delivered accordingly.
Apologies for any convenience this may have caused.

We would like to inform you that all WDF files have now been delivered. We apologise sincerely for the delay.

The SSIM and WDF productions are now running and being delivered. However, it is important to note that there is a latency of a few hours in the delivery process. Please bear with us as we work to expedite the SSIM and WDF productions, aiming to minimize the latency with delivery.

Additionally, FI Direct has now been updated and is once again operational.

We are pleased to announce that subsequent to Microsoft's update at approximately 08:43 GMT, we increased our resources to effectively manage the backlog resulting from the outage. Events are currently being distributed to customers' Event Hubs as anticipated. While there may be some latency during the distribution process, we anticipate the service to resume to normalcy shortly.

At 08:53, OAG observed a stabilised service from Microsoft. Subsequently, we increased our resource levels, and the current progress indicates that we are successfully clearing the backlog at an approximate rate of 120 - 150k messages per minute.

With this increased efficiency, we anticipate a swift resolution and foresee catching up on the backlog in the near term. We are monitoring the situation to ensure continued stability and optimal performance.

Microsoft are still in the midst of mitigating the issue and have advised that the scale-out and load balancing operations are progressing well, and success rates have significantly increased in the past 30 minutes. They will continue to monitor progression and provide an update as events warrant.

This incident has been resolved.

Update:

The final processing steps for the resolution of AA data for Dynamic Schedules is now complete and Dynamic Schedules will start publishing corrected AA Schedules.

Update :

Status data issues resolved.

Dynamic schedules issues are partially resolved with KL and DL recovered. The remaining fix for AA remains unresolved, but the recovery process is progressing well and expected to be fully resolved inside of the next 90 minutes.

Next update in 90 mins or when our final recovery process completes

Once again, apologies for this ongoing service disruption

Update:

For Status data, KL data has now been fixed and data will be flowing again for KL through the OAG toolset.

For Dynamic Schedules, we are continuing to work on this issue. For DL and KL the problems have now been resolved and the current estimate is AA data issues will be resolved within the next 90 minutes.

The next update will be in 90 minutes or sooner, if we can provide an update on resolution before this.

Apologies for the this ongoing service disruption.

Update:

For Status data products the impact is not with AA and DL as previously mentioned. Only KL is impacted. Remediation actions are ongoing and we expect this issue to be resolved within the next 90 minutes.

For Schedules Dynamic customers, the impact remains against AA, DL and KL. We expect resolution to take longer and we will provide further updates in 90 minutes in conjunction with our update on Status data.

Apologies for this ongoing service disruption. OAG are working hard to fix this issue as expeditiously as possible.