Batch Jobs queued up in US MTS

Incident Report for DataRobot

Resolved

A fix has been deployed to production and the issue is now resolved. All systems are operating normally.
Posted Jun 03, 2026 - 09:29 UTC

Update

We are continuing to monitor for any further issues.
Posted May 29, 2026 - 15:20 UTC

Monitoring

The remediation script has been deployed and Engineering is actively monitoring the situation. Batch jobs may take longer than usual to show as completed until a permanent fix is rolled out in the next production deployment.
Posted May 29, 2026 - 14:38 UTC

Update

The affected jobs have been resolved and the issue is mitigated. Engineering is actively working on a permanent fix and testing is currently underway
Posted May 29, 2026 - 10:22 UTC

Identified

Batch jobs that are currently in the queue are completed successfully; however, their completion status is not being updated correctly. As a temporary workaround, we are manually marking these jobs as completed while we work on implementing a solution.
Posted May 29, 2026 - 08:37 UTC
This incident affected: Managed AI Cloud (Website, API, Predictions, AutoML, AI Catalog and Data Ingest, AI Apps, MLOps, Pipeline, Notebooks, Generative AI LLM Playground, Generative AI VDB Builder).