The script successfully completed its run at 20:14 UTC, Thursday, 22 July. In the end, we reverted the resource resolution URLs of 520,512 DOIs. We identified over 1.1 million potentially affected DOIs in our review. The other ~600,000 DOIs that were not reverted as a part of our reversion script were not reverted because the original files provided to us included errors (e.g., typos in the DOIs - many of the DOIs had never been registered and therefore could not have their resource resolution URLs reverted).
Posted Jul 23, 2021 - 15:05 UTC
Update
We've now processed all of the affected DOIs. We have reverted all of the inadvertently updated resource resolution URLs. The script completed its run at 20:14 UTC, Thursday, 22 July.
Posted Jul 22, 2021 - 20:35 UTC
Update
We're nearly 48 hours into our run in production to revert the resource resolution URLs of the DOIs affected by this issue. There are still no errors, and we have processed over 706,000 DOIs (well over half). We're on track to have all affected DOIs reverted by late Thursday/early Friday of this week.
Posted Jul 21, 2021 - 18:49 UTC
Update
We're over 24 hours into our run in production to revert the resource resolution URLs of the DOIs affected by this issue. So far we haven't had any problems and have processed over 320,000 DOIs. Based on this rate, all affected DOIs should be reverted by late Thursday/early Friday of this week.
Posted Jul 20, 2021 - 21:31 UTC
Update
We have begun reverting the resource resolution URLs of the DOIs affected by this issue.
Posted Jul 19, 2021 - 20:50 UTC
Update
Good news! Our third full test run of our script is error free. We will begin reverting resource resolution URLs for the 22 members affected by this error starting today.
Posted Jul 19, 2021 - 19:02 UTC
Update
We have completed two test runs to revert the resource resolution URLs of the DOIs affected by this mistake. Both tests have gone well, but turned up problems with our reversion script. We're currently running a third test run and will begin reviewing the results of that test later today. While we'd love to get the reversions to production resource resolution URLs kicked off as soon as possible and revert all of the mistakes now, we have decided that running additional tests is the best option for all affected members. Thus, we will not begin the reversion on production resource resolution URLs until Monday, 19 July at the earliest.
Posted Jul 16, 2021 - 18:28 UTC
Update
We are currently going through the results of our test run to revert all of the affected DOIs' resource resolution URLs to ensure there are no errors in our script. If that review goes smoothly, we should be able to begin the reversions in production starting tomorrow.
Posted Jul 15, 2021 - 19:33 UTC
Update
Our technical team continues to work on testing the URL reversion script to ensure that it is making the desired reversions and only the desired reversions to the affected content. We'll begin reviewing test results as a team tomorrow and hope to kick off the resource resolution URL reversions of the affected DOIs on Friday, 16 July.
Posted Jul 14, 2021 - 17:17 UTC
Update
Our technical team will begin testing their script to revert the affected resource resolution URL later today. We're still looking at the second half of the week before we begin making URL reversions. We have also confirmed the list of all affected DOIs and will be sharing that with the 22 affected members shortly.
Posted Jul 13, 2021 - 16:39 UTC
Update
Our technical team is continuing to work on a fix. We're anticipating reverting the incorrect resource resolution URLs of the over 1.1 million affected DOIs during the second half of this week. This is the first time we're attempting a reversion of resource resolution URLs, so we're taking our time to ensure we get it right.
Posted Jul 12, 2021 - 14:53 UTC
Identified
In the end, to be safe, we're reverting the attempted updates of over 1.1 million DOIs as a part of this large batch of URL updates. We have contacted the 22 members who were affected by this mistake, and have restored the correct ownership of all DOIs that were inadvertently transferred. The next and final step for this issue is to revert all of the URLs that were mistakenly updated to their former resource resolution URL. This type of correction is a first for our technical team, so it is taking them time to create a script to correct the affected DOIs' URLs. We'll provide more updates next week.
Posted Jul 09, 2021 - 21:29 UTC
Investigating
There has been a mistake in processing a large batch of URL updates. Around 600,000 records owned by 22 members may have incorrectly been transferred from one member to another member. Members cannot update other members’ records; we did this internally. We are investigating who the affected members are and looking at options to revert the erroneous transfers. In the meantime, records in Metadata Search and our APIs may be giving false ownership data. More information here as we have it and we will contact the individual members when they have been identified.
Posted Jul 09, 2021 - 14:12 UTC
This incident affected: APIs (Public REST API, Polite REST API, OAI-PMH, XML API), Metadata Plus (Plus REST API, Plus OAI-PMH, XML Snapshots, JSON Snapshots), and Sites (Metadata search).