11 Apr 2017
11:54 UTC
Tom Hill
Two things are going to happen in response to the behaviour of this router, earlier this morning:
We need to fix some of the monitoring we have, and ensure that it doesn't break silently again.
We'll need to schedule an emergency software upgrade of this router for later this evening. This can be done gracefully, so customers won't be affected in the same manner. However, there are some EoMPLS tunnels terminated here, and those will have some form of outage (being point-to-point). We'll email all of those customers with the required details as soon as possible.
11 Apr 2017
10:17 UTC
Tom Hill
Correction: this outage was caused by a router failure on our part, rather than the singular connectivity provider.
We're still working on diagnosing the exact fault, but it does now also seem that the loss of connectivity did in fact cause signalling failures with the Cloud platform in York, which lead to an outage of VMs around 35-40 minutes later.
A further update will be issued to this status post, regarding our remedial action on the router side of things. Further diagnosis of the Cloud platform outage (with reference to this issue) will occur in the relevant status post.
My apologies for the initial confusion that my last update may have caused.
Tom
11 Apr 2017
09:56 UTC
Tom Hill
The second outage described around 09:28 (UTC+1) was in fact a red herring, and was part of the Cloud platform issues in York (reported upon separately).
The initial network interruption just before 09:00 (UTC+1) was related to a single network path, between Manchester & London, going down and coming back a few minutes part.
Only services hosted in Manchester, and reached from London, will have been affected by this particular event.
The connectivity provider that we utilise along this path will be grilled for more information, as we've not seen any official reports back from them.
As for now, this status will be resolved to remove confusion with the other ongoing work with our Cloud services in York.
11 Apr 2017
08:49 UTC
David Edwards
This morning between 08:55 and 08:59, some customers have reported a loss of network connectivity. We're looking into the causes now, and will update when we have more information.
A second brief outage was reported around 09:28 - we'll have more information when it arrives. All services should be back in operation, any customers with outstanding issues should give us a call, or raise a ticket.
11 Apr 2017
08:07 UTC
David Edwards
This morning between 08:55 and 08:59, some customers have reported a loss of network connectivity. We're looking into the causes now, and will update when we have more information.
All services should be back in operation, any customers with outstanding issues should give us a call, or raise a ticket.