Service Status

14 Feb 2018
18:16 UTC

Tom Hill

Hello,

Our provider have completed their testing and assert that it is not their hardware at fault. There has been some cleaning of inter-rack cabling in the mean time, so we've been asked to put our traffic back over the offending link, and monitor for further occurrences of frame errors.

All Manchester-York traffic is now back to flowing through the primary path, via Leeds.

Tom

14 Feb 2018
12:18 UTC

Tom Hill

Hello,

Further intrusive testing of the of the erroring circuit between Manchester and Leeds has lead suspicion to the inter-rack patching between ourselves and the circuit provider. As this patching is provided by the third party data centre operator, they are required to check and clean this.

As they are a third party, and because our two circuits are installed side-by-side, we're going to temporarily de-preference both of the links between Manchester and Leeds, making the primary path for Leeds/York to Manchester, go via London, adding some latency, but with no other impact (as it was at the beginning of this issue, in fact).

Hopefully this check of the inter-rack cabling won't take more than a few hours to resolve.

Thanks for your patience,

Tom

Expected resolution time updated: 14 Feb 2018, 23:30 UTC

9 Feb 2018
18:13 UTC

Tom Hill

Hello,

Intrusive testing upon the supplier's WDM equipment isolated the errors to the Leeds side of the circuit. To determine which part is at fault is now waiting on an on-site visit there to test those components individually.

We have re-balanced traffic back across both links for now, and will that for the weekend. Further updates to come on Monday as we receive the testing schedule.

Kind regards,

Tom

Expected resolution time updated: 13 Feb 2018, 23:30 UTC

8 Feb 2018
13:51 UTC

Tom Hill

Hello,

Pursuant to the minor error rate still being encountered on one of the two links between Leeds and Manchester, we will at 14:00 (UTC+0) allow our provider to take down one link in order to perform intrusive end-to-end testing. It is hoped that this will pinpoint the cause of the lingering frame error rate.

This update is simply to keep everyone appraised of our progress, as testing will not impact any traffic around the Bytemark network; traffic will flow as normal between Leeds and Manchester, albeit with reduced capacity. The testing itself will not take long (1-2hrs) and may be extended if there arises a need to swap-out hardware.

Kind regards,

Tom

Expected resolution time updated: 9 Feb 2018, 23:30 UTC

17 Jan 2018
00:17 UTC

Tom Hill

Hello,

Following some remedial work in Leeds, our supplier have given us the all clear to begin re-using our links between Manchester and Leeds.

It appears that whilst one of the links is functioning without error, the other is still occasionally dropping a packet, circa 1pps. With half of our Trans-Pennine traffic balanced across it, this is likely to be an acceptable level for TCP error correction.

However, we will continue to debug this with our supplier as it is not acceptable within the longer term.

For now, traffic is flowing directly between Manchester & Leeds once more. If you find specific issues with network traffic across this path, please do contact support.

We will endeavour to update this status with further progress in the near future.

Tom