Due to a fire in our main datacenter, most of the services hosted by Humanities IT Services are currently down.
For information on the status of other UU services, please visit the liveblog page on uu.nl.
We're working on our plan for restoring full services once our datacenter is back online. We've reached out to our service owners to figure out which services need to be prioritized.
In addition, we're handling some urgent cases to restore additional emergency services.
We are still waiting for power to be restored to the datacenter, which should be on Wednesday. Once we have power, we (and ITS) will need to verify that our hardware is working properly and then begin working on restoring our core services.
We're still dealing with a number of unknowns, so we are hesitant to provide any kind of timeline. Once we've restored initial core services, we'll be able to provide a more detailed timeline.
If you have any questions or concerns, please contact us at our regular email address. Please note that our team is busy, so we may not be able to respond to your email immediately.
ITS has managed to resolve the DNS issue, which allows us to bring up several emergency landing pages and other services.
Most sites in our wordpress environments should now be redirected to this page, while some others will be directed to dedicated info pages for those sites.
We've been able to get the first few emergency pages up and running (most importantly, diergeneeskunde.nl) and deployed servers for other critical services.
However, we're still blocked by a DNS issue. We hope that ITS will be able to resolve it by tomorrow morning at the latest.
We're currently hard at work coordinating our recovery plan internally and with ITS; at this point it seems likely that we won't be able to start full recovery until Thursday at the earliest. This is due to delays in restoring power at the datacenter.
In the meantime we're also working on deploying temporary infrastructure to get critical services back up, as well as a general status page for the UU wordpress environment.
However, we're currently blocked by a DNS issue that ITS will have to resolve before we can proceed with bringing the temporary infrastructure online.
We've received word that NorthC is not going to be able to restore power to our equipment tomorrow.
We will evaluate this news tomorrow and see what can be done in the meantime.
NorthC has indicated that it may take 72 hours to restore power. This means that we may be able to start bringing up infrastructure again on Monday at the earliest. However, full restoration will take longer.
As such, we still cannot give a concrete timeline at this point. We hope to have a more detailed update on Monday.
In the meantime, we have been unable to get the temporary landing page running due to unforseen issues outside our control. We're hoping to get this resolved soon, but we can make no promises at this point.
We have received word from ITS that they've visited the datacenter. While the building is still without power, our hardware does not appear to be affected by the fire.
We are still waiting on NorthC to provide us with a timeline for when power is restored and when we can start starting our servers.
We have started working on restoring critical (internal) services in our alternate datacenter.
This will include a temporary landing page for users informing them of the outage.
The fire department has declared 'brand meester', which means the fire is under control and is no longer a risk of spreading to other sections of the building.
We have confirmed that our off-site backups are present and accounted for. We expect little to no data-loss to occur once our servers are restored.
On May 7th, around 8:30, a fire broke out in one of the technical rooms in the NorthC datacenter in Almere, where most of our servers are located. By order of the fire department, power was cut off to the datacenter around 9:10.
As a result, most of our services went offline.