16 Apr 2018
23:36 UTC
Tom Hill
Everything looks good. We're now leaving site.
If you were watching, thank you very much for your patience.
Should any outages occur after this point, please ensure that you raise us via the support-urgent methods as per usual.
Kind regards,
Tom
16 Apr 2018
22:46 UTC
Tom Hill
Hello,
We're done. All optical transceivers have now been replaced with new items.
We'll be closely monitoring the situation over the next hour.
Kind regards,
Tom
16 Apr 2018
22:26 UTC
Tom Hill
Hello,
We're half-way through the distribution switches. Beginning on the second of the two shortly.
Tom
16 Apr 2018
22:02 UTC
Tom Hill
Hello,
The two core switches have now had all of their transceivers replaced, without incident.
We'll be moving on to the distribution devices now, which deal with access traffic. This is a little more involved, but as before we are not expecting any interruptions.
Tom
16 Apr 2018
21:31 UTC
Tom Hill
Hello,
We're starting the transceiver replacements now, beginning with the two core switches, which underpin inter-rack connectivity for the storage LANs.
Tom
16 Apr 2018
21:30 UTC
Tom Hill
Hello,
A timely reminder that this work will begin in approximately one hour. We will be monitoring support channels, as well as our IRC channel during the work, and periodic updates will be made to this post.
As previously noted, this work is expected to be undertaken without an interruption to your services.
Kind regards,
Tom
16 Apr 2018
21:30 UTC
Tom Hill
Hello,
In reference to the recent update made by the Operations Director, we have received a quantity of replacement optical transceivers - manufactured by a different vendor - and will be installing these tonight during an emergency maintenance window.
Our aim is to perform this work without service interruption, and with every precaution to avoid causing significant disruption to network traffic. There is, however, some risk in such invasive network surgery, and during this period the Manchester data centre will temporarily be running with reduced resilience to connectivity outages.
We appreciate that this announcement comes with very short notice, but under the circumstances of recent events, we feel it is imperative that we conduct this remedial work as soon as possible. Both our National Core & the York Data Centre networks will be unaffected by this work.
We hope that our efforts tonight will go towards improving the reliability of the entire platform in Manchester, and we would like to take this opportunity to thank you for your continued patience whilst we work to resolve the issues identified with our hardware.
If you have any queries pertaining to this work, please feel free to reach out to me (FAO Tom) via the usual support avenues. As usual, pertinent updates will be made to this thread to keep you abreast of our progress.
Kind regards,
Tom Hill Network Manager
Expected resolution time updated: 16 Apr 2018, 23:00 UTC