[AT RISK] Transceiver replacements, Manchester

Expected resolution: 16 Apr 2018, 23:00 UTC
Return to issues
Issue status: Resolved Date:

16 Apr 2018
23:36 UTC

Posted by:

Tom Hill

Everything looks good. We're now leaving site.

If you were watching, thank you very much for your patience.

Should any outages occur after this point, please ensure that you raise us via the support-urgent methods as per usual.

Kind regards,

Tom

Issue status: In Progress Date:

16 Apr 2018
22:46 UTC

Posted by:

Tom Hill

Hello,

We're done. All optical transceivers have now been replaced with new items.

We'll be closely monitoring the situation over the next hour.

Kind regards,

Tom

Issue status: In Progress Date:

16 Apr 2018
22:26 UTC

Posted by:

Tom Hill

Hello,

We're half-way through the distribution switches. Beginning on the second of the two shortly.

Tom

Issue status: In Progress Date:

16 Apr 2018
22:02 UTC

Posted by:

Tom Hill

Hello,

The two core switches have now had all of their transceivers replaced, without incident.

We'll be moving on to the distribution devices now, which deal with access traffic. This is a little more involved, but as before we are not expecting any interruptions.

Tom

Issue status: In Progress Date:

16 Apr 2018
21:31 UTC

Posted by:

Tom Hill

Hello,

We're starting the transceiver replacements now, beginning with the two core switches, which underpin inter-rack connectivity for the storage LANs.

Tom

Issue status: Scheduled Date:

16 Apr 2018
21:30 UTC

Posted by:

Tom Hill

Hello,

A timely reminder that this work will begin in approximately one hour. We will be monitoring support channels, as well as our IRC channel during the work, and periodic updates will be made to this post.

As previously noted, this work is expected to be undertaken without an interruption to your services.

Kind regards,

Tom

Issue status: Scheduled Date:

16 Apr 2018
21:30 UTC

Posted by:

Tom Hill

Hello,

In reference to the recent update made by the Operations Director, we have received a quantity of replacement optical transceivers - manufactured by a different vendor - and will be installing these tonight during an emergency maintenance window.

Our aim is to perform this work without service interruption, and with every precaution to avoid causing significant disruption to network traffic. There is, however, some risk in such invasive network surgery, and during this period the Manchester data centre will temporarily be running with reduced resilience to connectivity outages.

We appreciate that this announcement comes with very short notice, but under the circumstances of recent events, we feel it is imperative that we conduct this remedial work as soon as possible. Both our National Core & the York Data Centre networks will be unaffected by this work.

We hope that our efforts tonight will go towards improving the reliability of the entire platform in Manchester, and we would like to take this opportunity to thank you for your continued patience whilst we work to resolve the issues identified with our hardware.

If you have any queries pertaining to this work, please feel free to reach out to me (FAO Tom) via the usual support avenues. As usual, pertinent updates will be made to this thread to keep you abreast of our progress.

Kind regards,

Tom Hill Network Manager

Expected resolution time updated: 16 Apr 2018, 23:00 UTC

Return to issues

Issue still not addressed? Please contact support.