All systems are reporting healthy again, so we're slowly transitioning back to normal operations. If you are still experiencing issues, please contact us and we'll investigate.
We're also going to be taking some rest time (in shifts), so we'll be understaffed in the next few days. Thus, it may still take a little longer to respond to all e-mails.
A fix for the performance issues has been found. ITS is currently applying it to all affected servers. We expect this to be done by the end of the day.
In collaboration with ITS we've identified the source of the performance issues. They are currently working on a solution.
We've restored most services; we've prioritized the services that were on our priority list from last week, so we might have overlooked some others. We'll do a full check tomorrow. Thanks for your patience!
Due to a temporary power failure in the datacenter, most services are currently offline. We have no clear picture on the full impact, nor the path to restoration, at this point.
We've restored the services on our list for this week.
We are seeing some severe performance issues across the board. We've done what we can to mitigate them, but the cause seems to be somewhere in the central IT systems. We'll update you as soon as we have more information.
All priority services and most of the other services are back online. Others will be restored as soon as possible. We're prioritizing public-facing services first, so non-critical developer/admin services might be delayed till early next week.
We have seen that some servers are a bit slower than normal, we're also looking into this.
We've been able to restore our core services ahead of schedule.
Some services may appear to be working, but we haven't been able to thoroughly test them yet. We will post an update as soon as we can; but in the meantime we kindly ask everyone to be patient and wait till we checked that everything is working as expected.
Our colleagues at ITS are currently hard at work with the initial startup of the UU infrastructure at the datacenter.
We've discussed our plans for tomorrow with ITS and are in contact with them about their progress and the next steps for us. We're still expecting to start working on bringing services online tomorrow.
We cannot guarantee our services will work reliably when they first start up, so we kindly ask everyone to be patient and wait till we and/or ITS confirm that everything is working as expected.
NorthC has restored the power in the datacenter itself a little ahead of schedule. For now, this has no impact on our timeline.
We're currently preparing for tomorrow, and have decided to stop any other recovery efforts for now. This is to properly plan out final details and preserve our staff for the long few days ahead.