Replacement of Core Router, Manchester

Expected resolution: 25 Nov 2015, 12:30 UTC
Return to issues
Issue status: Resolved Date:

25 Nov 2015
12:24 UTC

Posted by:

Tom Hill

And we're all done! All ports, prefixes and paths should all be back to where they were before the maintenance.

We'll of course be keeping an eye on everything throughout the rest of the day,

Issue status: Investigating Date:

25 Nov 2015
11:51 UTC

Posted by:

Tom Hill

More delays finishing this work off completely - we're having some trouble getting OSPF to work with our data centre routers.

All other work has been complete for a little while, and the new router is forwarding quite a bit of traffic with seemingly no adverse affect.

Issue status: Investigating Date:

25 Nov 2015
10:19 UTC

Posted by:

Tom Hill

The vast majority of services should return shortly. There's one job left to do to ensure that the Manchester DC is fully resilient once again. We're overrunning, but this is mainly due to the unexpected fault that occurred earlier.

Issue status: Investigating Date:

25 Nov 2015
09:07 UTC

Posted by:

Tom Hill

The new router is in place in the rack, and connections will be coming back shortly. I will start with the core router ports, and move on to customer-facing ports last.

Issue status: Investigating Date:

25 Nov 2015
08:08 UTC

Posted by:

Tom Hill

There was a missing OSPFv3 interface statement in Manchester; very little actually relies upon this (almost all routes are exchanged via BGP) so I'm fairly sure this was mostly down to connectivity to our router loopbacks, and potentially a firewall linknet or two.

If you were affected by this disruption to IPv6 in Manchester, please do let us know via support ticket.

Issue status: Investigating Date:

25 Nov 2015
08:01 UTC

Posted by:

Tom Hill

We've noticed that some IPv6 OSPF routes are failing. This is causing some sporadic IPv6 reachability issues; we're working to resolve this immediately.

Issue status: Resolved Date:

25 Nov 2015
07:21 UTC

Posted by:

Tom Hill

IP traffic is being moved away from the router, by preference. I'm just waiting for this to settle down, and then the router will be turned off.

Issue status: Investigating Date:

25 Nov 2015
06:58 UTC

Posted by:

Tom Hill

This maintenance will be beginning in the next 5-10 minutes.

Issue status: Investigating Date:

24 Nov 2015
19:01 UTC

Posted by:

Tom Hill

This coming Wednesday morning, between 07:00 and 10:00 (UTC+0), we will be replacing one of the core routers in Manchester.

As all of the core network is transparently resilient from the perspective of the data centres, customers with services hosted in Manchester (Reynolds House) should not notice any untoward loss of service. However, this should be considered an 'at-risk' period due to the lack of core network resilience whilst we swap the router out.

Any customers with network services directly attached to the router in question will see an outage of those services, and will have been notified a week in advance by way of a ticket from our support system.

For the very vigilant customers, you may notice the following whilst this router is being swapped out:

  • Traffic between Manchester and London (so via LINX/LONAP) will run via York (+2ms RTT)
  • Our IP transit announcements to GT-T (AS 3257) will be withdrawn (leaving NTT & Cogent)
  • cr1.man will disappear, and cr4.man will appear in its place (pending PTR record propagation!)

Timely updates will be provided here as we continue through the maintenance. Please do not hesitate to get in touch with the support team if you have any further queries as regards this work.

Many thanks,

Tom

Return to issues

Issue still not addressed? Please contact support.