Metadata indexing failure and instability on the REST API (Starting Friday 12th 20:00 UTC and continuing) affecting several other tools

Incident Report for Crossref

Resolved

We have updated the service that synchronises new and updated metadata records with the REST API; all three pools of the REST API are performing well as a result (as are the services that run on it). Thus, we are resolving this incident. -IF
Posted Oct 02, 2025 - 15:31 UTC

Update

We've identified and fixed one issue which was affecting citation counts. We're still working on two additional tasks: upgrading Elasticsearch in the REST API and updating the service that synchronises new and updated metadata records with the REST API. -GH
Posted Sep 26, 2025 - 13:59 UTC

Identified

The public and polite pools are still experiencing some instability, which is affecting downstream tools such as Participation Reports, Search, and Crossmark. We're working on a fix incl. infrastructure upgrades that should help long-term.
-GH
Posted Sep 23, 2025 - 13:26 UTC

Update

We are continuing to monitor for any further issues.
Posted Sep 22, 2025 - 19:08 UTC

Update

The public and polite pools are still showing as unstable with long response times and timeouts occurring. We are still looking into this as a priority and will post with more information as we know it.
Posted Sep 19, 2025 - 10:15 UTC

Monitoring

A fix has been implemented and we are monitoring the results. Do to the heavy workload of indexing a large volume of records yesterday, we have observed longer response times in the Public and Polite pools of the REST API. We believe those response times should improve throughout the day. -IF
Posted Sep 17, 2025 - 12:48 UTC

Update

Our technical team is re-pushing all of the records that have failed to be indexed since Friday, 12 September. Those records should be indexed in the REST API in the next 24 hours. -IF
Posted Sep 16, 2025 - 14:14 UTC

Update

We are continuing to investigate this issue.
Posted Sep 15, 2025 - 16:33 UTC

Investigating

We have found that some records failed to be indexed by the REST API on Friday 12th September, and continuing. Our technical team is investigating.
Posted Sep 15, 2025 - 09:13 UTC
This incident affected: APIs (Public REST API, Polite REST API, Crossmark dialog server) and Sites (Metadata search, Participation reports).