Postmortem - Estimate Submitted Trigger Configuration Issue

Services impacted

• Event-based integration processing (webhook runs)

• Callback service

• Production transaction flow

Incident summary

On June 1, 2026, production transaction processing halted again due to the same class of configuration issue previously identified in the May 30 incident. Escalated monitoring implemented after the prior incident allowed the engineering team to detect the disruption at the time of occurrence. A second integration containing incompatible configuration metadata was identified as the contributing cause. The team used this recurrence to complete a full root cause investigation and deploy a permanent fix across all affected integrations.

Root cause

A second integration containing an unsupported trigger type in its configuration metadata caused the configuration service to throw mapping errors, disabling the callback service and stalling all event-driven integration runs. The weekend resolution of the prior incident addressed only immediate recovery without eliminating the underlying configuration data. Root cause analysis confirmed that four integrations in total contained this incompatible configuration, and the system lacked both pre-activation validation and front-end controls to prevent such configurations from being activated.

Resolution

The second affected integration was identified and disabled at 9:15 AM CT on June 1, 2026. A full root cause investigation was completed by 12:19 PM CT, confirmed through reproduction in a non-production environment. A database patch was applied at 1:15 PM CT to remove the incompatible trigger type from all four affected integrations. Successful deployment to the production environment was confirmed at 1:30 PM CT.

Corrective actions

1. Removed the incompatible trigger type from all four affected integrations via database patch. (Completed — June 1, 2026)

2. Expanded monitoring to cover all job types, including event pickup backlogs, to enable earlier detection of processing delays. (Completed — June 3, 2026)

3. Removed the unsupported trigger type from the webhook configuration template to prevent future use. (In progress — ETA June 4, 2026)

4. Implemented pre-deployment validation to check integration configuration compatibility with the current configuration service build before release. (In progress — ETA June 5, 2026; next deployment planned June 11, 2026)

5. Implementing front-end validation to prevent activation of integrations with incompatible configuration metadata. (In progress — ETA TBD)