Update to 9th November Maintenance

Published on 14 Nov 2016

On Wednesday 9th November we undertook some maintenance on our firewall infrastructure to enable our new office security VPN.
We anticipated no effect on services, but took the precautionary measure of stationing an engineer on site at the hosting centre in case of issues.
In the event, while the switch of configuration occurred with little issue on the firewalls, we quickly discovered that the hypervisor software we use contained a bug which was not able to cope with the change in the underlying network stack.

As a result some of our live systems could no longer be contacted.
We worked quickly to identify the problem and restart the affected systems. We had scheduled the work out of business hours and so believe that we limited the customer impact.

The network change we implemented was not one that occurs regularly and so the conditions for the failure should not be reproduced in the foreseeable future .
If any of our customers were affected we sincerely apologise and are looking at the cause in order to prevent a similar event occurring again.