4 Dec 2015
08:42 UTC
Ian Chilton
All looks ok now so i'm resolving this. We'll debug the network issue internally.
4 Dec 2015
08:35 UTC
Ian Chilton
I appear to have tracked this down to packet loss on a storage network between some of the heads and the tail. I've failed over to a backup link and things are instantly looking much better. Continuing to monitor.
4 Dec 2015
07:50 UTC
Ian Chilton
Have established that this doesn't appear to be a problem with the tail after all. Looking in to head53 now.
4 Dec 2015
07:34 UTC
Ian Chilton
We're digging in to this - will keep this issue updated.
4 Dec 2015
07:22 UTC
Telyn Roat
We've discovered an issue with BigV tail51 - this is a SATA storage tail for VMs in the York BigV zone. Affected VMs show slow I/O with high IOwait to any SATA disks on that tail.
We're currently raising our BigV operations staff to investigate and assist.