Cloud Services Brain Currently Experiencing Problems

Expected resolution: We are currently investigating.
Return to issues
Issue status: Resolved Date:

5 Aug 2016
15:37 UTC

Posted by:

Bytemark Engineer

The Bytemark Cloud brain uses a redundant synchronous pair of database servers. On the night of this incident, the PostgreSQL slave was required to reconnect to the master due to an error, however due to an earlier PostgreSQL client version change made during routine maintenance this server was unable to connect to the master database server because of SSL incompatibilities. To preserve data integrity, writes on the master are disallowed when no synchronous slaves are connected, which caused the Bytemark Cloud API to become unavailable.

Once this issue was identified and the slave reconnected, database writes were allowed to proceed and the brain resumed normal operations.

To prevent this kind of incident happening in the future a second database slave is being commissioned. This will remove the single-point-of-failure which currently exists due to having only one synchronous slave.

Issue status: Resolved Date:

2 Aug 2016
08:29 UTC

Posted by:

Paul Cammish

A database issue was identified and resolved, and service should have returned to normal.

Customers should be able to start, stop and modify servers again normally.

If you are still experiencing any issues with your Cloud Server, please don't hesitate to contact support.

Issue status: Investigating Date:

2 Aug 2016
01:13 UTC

Posted by:

Bytemark Engineer

The Bytemark Cloud Service brain component is currently experiencing problems.

This will not affect any running virtual machines, but will prevent administrative services such as: * Stopping/starting VMs * Creating/deleting VMs * Adding/resizing disks

We are looking into this and will update this status post accordingly.

Return to issues

Issue still not addressed? Please contact support.