We replaced the affected hardware on 5 April and have been monitoring the replacement since. Luckily, there have been no issues with the new hardware since. Thus, this incident has been resolved. All affected services are fully operational.
Posted Apr 16, 2019 - 16:36 UTC
Update
We've been unable to get the affected machine back to full health, so we're working on a replacement. We'll keep you updated as to our progress. Metadata Search performance remains degraded.
Posted Mar 29, 2019 - 20:34 UTC
Update
We have more work to do to resolve these hardware issues, but we're done for today. Performance of our metadata search is still degraded. All other services are operational.
Posted Mar 29, 2019 - 01:51 UTC
Update
We're continuing our work on the hardware issues from yesterday. Our efforts to move affected systems to auxiliary hardware while we troubleshoot has resulted in a stable day for our services. That said, more work continues and users may experience degraded performance as a result. We're monitoring.
Posted Mar 28, 2019 - 20:45 UTC
Update
We're still working to get this fully resolved. Users may experience degraded performance from Crossref Metadata Search, as we continue or work.
Posted Mar 27, 2019 - 18:54 UTC
Monitoring
A temporary fix has been implemented - affected systems have been moved to auxiliary hardware - and are currently operational. We're monitoring and will be performing additional work later today, so users may experience additional disruptions as we work toward a permanent fix.
Posted Mar 27, 2019 - 17:09 UTC
Identified
The issue has been identified and we are working on a fix, as part of this fix we are having to redirect some of our services. Users may experience degraded performance with the deposit system, Metadata Manager, and our APIs.
Posted Mar 27, 2019 - 14:28 UTC
Update
We are continuing to investigate this issue.
Posted Mar 27, 2019 - 09:36 UTC
Investigating
The hardware problems we experienced yesterday are continuing. The deposit system may be unstable and/or working in a degraded state until we are able to fix or replace the hardware.
Posted Mar 27, 2019 - 09:17 UTC
This incident affected: Meta (deliberately-unreliable server), Content Registration (Admin tool), Metadata Plus (Plus OAI-PMH), Sites (Metadata search), Beta (Metadata Manager), and APIs (XML API, OpenURL).