29 Jun 2017
21:56 UTC
Peter Taphouse
As of around 14:00 today, we've started getting alerts that some Cloud Servers are experiencing problems with disk IO for one of the storage pools on tail56.
This will manifest as either very slow disk throughput on SATA/SSD storage, or simply problems accessing files, and may cause processes or servers to crash, and affected servers may fail to restart.
All affected disks are being migrated to other tails at the moment, and affected servers should return to normal as they are moved.
We will will update this post as soon once there are further updates.
Update at 14:51 - So far 25% of the disks have been migrated
Update at 15:18 - Over 50% of the disks have now been migrated.
Update at 15:44 - The remaining 30% of disks are being migrated.
Update at 16:57 - There are now only 3 disks left to move which are being migrated now - we expect these to complete shortly, and we are manually checking for any affected machines which are not responding to pings.
Updated at 22:55 - All disk migrations completed before 17:15 and checks on any affected VMs are complete.
29 Jun 2017
15:59 UTC
Paul Cammish
As of around 14:00 today, we've started getting alerts that some Cloud Servers are experiencing problems with disk IO for one of the storage pools on tail56.
This will manifest as either very slow disk throughput on SATA/SSD storage, or simply problems accessing files, and may cause processes or servers to crash, and affected servers may fail to restart.
All affected disks are being migrated to other tails at the moment, and affected servers should return to normal as they are moved.
We will will update this post as soon once there are further updates.
Update at 14:51 - So far 25% of the disks have been migrated
Update at 15:18 - Over 50% of the disks have now been migrated.
Update at 15:44 - The remaining 30% of disks are being migrated.
Update at 16:57 - There are now only 3 disks left to move which are being migrated now - we expect these to complete shortly, and we are manually checking for any affected machines which are not responding to pings.