25 Nov 2015
12:24 UTC
Tom Hill
And we're all done! All ports, prefixes and paths should all be back to where they were before the maintenance.
We'll of course be keeping an eye on everything throughout the rest of the day,
25 Nov 2015
11:51 UTC
Tom Hill
More delays finishing this work off completely - we're having some trouble getting OSPF to work with our data centre routers.
All other work has been complete for a little while, and the new router is forwarding quite a bit of traffic with seemingly no adverse affect.
25 Nov 2015
10:19 UTC
Tom Hill
The vast majority of services should return shortly. There's one job left to do to ensure that the Manchester DC is fully resilient once again. We're overrunning, but this is mainly due to the unexpected fault that occurred earlier.
25 Nov 2015
09:07 UTC
Tom Hill
The new router is in place in the rack, and connections will be coming back shortly. I will start with the core router ports, and move on to customer-facing ports last.
25 Nov 2015
08:08 UTC
Tom Hill
There was a missing OSPFv3 interface statement in Manchester; very little actually relies upon this (almost all routes are exchanged via BGP) so I'm fairly sure this was mostly down to connectivity to our router loopbacks, and potentially a firewall linknet or two.
If you were affected by this disruption to IPv6 in Manchester, please do let us know via support ticket.
25 Nov 2015
08:01 UTC
Tom Hill
We've noticed that some IPv6 OSPF routes are failing. This is causing some sporadic IPv6 reachability issues; we're working to resolve this immediately.
25 Nov 2015
07:21 UTC
Tom Hill
IP traffic is being moved away from the router, by preference. I'm just waiting for this to settle down, and then the router will be turned off.
25 Nov 2015
06:58 UTC
Tom Hill
This maintenance will be beginning in the next 5-10 minutes.
24 Nov 2015
19:01 UTC
Tom Hill
This coming Wednesday morning, between 07:00 and 10:00 (UTC+0), we will be replacing one of the core routers in Manchester.
As all of the core network is transparently resilient from the perspective of the data centres, customers with services hosted in Manchester (Reynolds House) should not notice any untoward loss of service. However, this should be considered an 'at-risk' period due to the lack of core network resilience whilst we swap the router out.
Any customers with network services directly attached to the router in question will see an outage of those services, and will have been notified a week in advance by way of a ticket from our support system.
For the very vigilant customers, you may notice the following whilst this router is being swapped out:
Timely updates will be provided here as we continue through the maintenance. Please do not hesitate to get in touch with the support team if you have any further queries as regards this work.
Many thanks,
Tom