Failed delivery of notification emails for very large query submissions and some forward-link alerts

Incident Report for Crossref

Resolved

This incident has been resolved. -IF
Posted Aug 27, 2025 - 13:16 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Aug 26, 2025 - 15:00 UTC

Update

We have confirmed that all notifications for registrations are being delivered and have been since 22 July. A small fraction of query submission email notifications (for very large queries) and forward-link alerts are going undelivered. Our technical team continues to work on a fix. -IF
Posted Aug 08, 2025 - 17:23 UTC

Update

We are continuing to work on a fix for this issue.
Posted Aug 06, 2025 - 18:27 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 30, 2025 - 16:07 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 25, 2025 - 20:41 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 23, 2025 - 20:20 UTC

Update

We have resolved the problems for the vast majority of members and those submissions, but we know that a fraction of notifications are failing for very large submissions. We are continuing to work on a fix for this issue. -IF
Posted Jul 22, 2025 - 19:39 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 21, 2025 - 12:42 UTC

Update

Yesterday's release did fix delivery of notification emails for many submissions, but we still have a small minority of notification emails that are going undelivered. Our technical team continues to work on a fix. -IF
Posted Jul 17, 2025 - 16:29 UTC

Update

We'll have an emergency, unplanned release (https://status.crossref.org/incidents/c5nyp41k6vx5) in ~20 minutes to address the ongoing errors with delivery of notification emails after registrations.

During that release window - from 1500 to 1700 UTC today - our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the emergency maintenance has been completed.
Posted Jul 16, 2025 - 14:40 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 15, 2025 - 20:51 UTC

Update

Delivery of email notifications of submission logs following a registration (for Simple Text Query form, web deposit form, and XML submissions) has been restored for many members and their submissions. We are working to remedy the other outstanding errors that are causing some email notifications to go undelivered. -IF
Posted Jul 14, 2025 - 17:42 UTC

Identified

Delivery of email notifications of submission logs following a registration (for Simple Text Query form, web deposit form, and XML submissions) began failing again late yesterday. Our technical team continues to work on a fix. -IF
Posted Jul 12, 2025 - 15:20 UTC

Update

Submissions from Metadata Manager and the Simple Text Query form have been restored. Unfortunately, delivery of email notifications of submission logs following a registration (for Simple Text Query form, web deposit form, and XML submissions) do appear to be delayed. We believe this is due to the volume of notification emails in the system, and the result of quite a backlog there. We are monitoring. -IF
Posted Jul 11, 2025 - 21:16 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 11, 2025 - 15:22 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jul 11, 2025 - 14:08 UTC

Update

We'll have an emergency, unplanned release (https://status.crossref.org/incidents/1c42q5j29ds2) in ~30 minutes to address the ongoing issues with Metadata Manager and newly reported errors with delivery of notification emails after registrations.

During that release window - from 1430 to 1630 today - our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the emergency maintenance has been completed.
Posted Jul 11, 2025 - 14:08 UTC

Update

Errors to Crossmark dialog boxes have been fixed, as you can see with this example: https://crossmark.crossref.org/dialog/?doi=10.36676/girt.v12.i1.109&domain=pdf

We're continuing to work on Metadata Manager and the Simple Text Query form. -IF
Posted Jul 10, 2025 - 20:36 UTC

Update

We are continuing to work on a fix for the errors present with Metadata Manager registrations. We have also identified errors with reference registrations via the Simple Text Query form (https://apps.crossref.org/SimpleTextQuery/) and errors in the display of Crossmark dialog boxes (e.g., https://crossmark.crossref.org/dialog/?doi=10.3233/JIFS-179309&domain=pdf). -IF
Posted Jul 10, 2025 - 19:21 UTC

Identified

The issue has been identified and we continue to work on a fix for Metadata Manager registrations that are failing with the 'TypeError' error messages. We have deployed a fix for the depositor reports. That issue is resolved (e.g., https://data.crossref.org/depositorreport?pubid=J458872). -IF
Posted Jul 10, 2025 - 14:08 UTC

Investigating

As a result of our recent AWS migration (https://status.crossref.org/incidents/s92nm4xn1w84), Metadata Manager submissions are currently returning 'TypeError' error messages and our depositor reports are returning a 404 'Not Found' status code error message. Our technical team is working on a fix. -IF
Posted Jul 10, 2025 - 03:19 UTC
This incident affected: Beta (Metadata Manager) and APIs (Crossmark dialog server).