5 Aug 2016
15:37 UTC
Bytemark Engineer
The Bytemark Cloud brain uses a redundant synchronous pair of database servers. On the night of this incident, the PostgreSQL slave was required to reconnect to the master due to an error, however due to an earlier PostgreSQL client version change made during routine maintenance this server was unable to connect to the master database server because of SSL incompatibilities. To preserve data integrity, writes on the master are disallowed when no synchronous slaves are connected, which caused the Bytemark Cloud API to become unavailable.
Once this issue was identified and the slave reconnected, database writes were allowed to proceed and the brain resumed normal operations.
To prevent this kind of incident happening in the future a second database slave is being commissioned. This will remove the single-point-of-failure which currently exists due to having only one synchronous slave.
2 Aug 2016
08:29 UTC
Paul Cammish
A database issue was identified and resolved, and service should have returned to normal.
Customers should be able to start, stop and modify servers again normally.
If you are still experiencing any issues with your Cloud Server, please don't hesitate to contact support.
2 Aug 2016
01:13 UTC
Bytemark Engineer
The Bytemark Cloud Service brain component is currently experiencing problems.
This will not affect any running virtual machines, but will prevent administrative services such as: * Stopping/starting VMs * Creating/deleting VMs * Adding/resizing disks
We are looking into this and will update this status post accordingly.