12 Apr 2017
00:56 UTC
Tom Hill
All traffic routing has been returned to normal following the router's software upgrade, which went swimmingly.
We'll be leaving site very shortly, so of course any trouble tickets should be directed to the urgent support address from here on.
Tom
12 Apr 2017
00:37 UTC
Tom Hill
The router upgrade was completed by 01:30, and Manchester once again has resilient routers & a direct path to London.
Further remedial work is to reinstate the GTT IP transit sessions, and reinstate link preferences facing the data centre.
Once this all looks to be in place, and running happily, we'll send a further update (and resolve the status) just before leaving site.
Tom
11 Apr 2017
23:38 UTC
Tom Hill
All of the necessary files are presently being loaded on to the router, which takes a fairly tedious chunk of time due to each one being checked for validity as it goes.
Just in case I'm updating the finish time to 01:30, but I will be aiming to be bringing traffic back before 01:15 - providing the upgrade continues satisfactorily.
11 Apr 2017
22:53 UTC
Tom Hill
Quick update: Customer traffic is all now drained from the affected router, and won't be going back to it until we've completed the upgrade.
Our GTT transit is connected to this router, and has now been removed from our transit mix. Again, this will not be returning until we're satisfied with the upgrade (Cogent, NTT & all peering sessions will remain unaffected).
We're now just in the process of increasing the cost on the Manchester to London connectivity, to avoid traffic from other parts of the network, using that backhaul.
Following that, we'll shutdown all interior BGP sessions, and upgrade the router.
11 Apr 2017
22:07 UTC
Tom Hill
Hello,
As discussed in an earlier (now resolved) issue from this morning, we will be performing and emergency upgrade of the router we hold in Kilburn House (part of Equinix MA1). The router (cr4.man) will have all customer traffic moved away gracefully, and will then be rebooted with the new software.
This is not expected to be the cause of any disruption, except in the cases where we have explicitly informed customers, notices for which went out earlier this afternoon.
Updates will be provided as we go, and we'll be paying attention to the main ticket queues, as well as all pertinent monitoring.
Thank you for your patience.
Tom