Postmortem - 01029964 - AM Region Latency and Intermittent Server Errors

Customer incident report

Service(s) impacted

• Corrigo Enterprise platform — intermittent "500 Internal Server Error" responses and severe latency

• 248 distinct customer environments across all AM region hosting farms, including some whose own servers were unaffected but were routed through the same shared infrastructure

Incident summary

On July 17, 2026, customers across the AM region experienced intermittent "500 Internal Server Error" pages and severe latency in the Corrigo Enterprise platform, affecting 248 distinct customer environments across all AM hosting farms. The disruption began at 20:49 UTC (3:49 PM CDT) and was fully resolved by 21:37 UTC (4:37 PM CDT), a total of approximately 48 minutes, including two brief windows of near-total service disruption. Approximately 2.4% of requests failed during the most affected 32-minute period, and no customer data was lost. Upon detection, our engineering and infrastructure teams immediately mobilized to identify the cause and restore service.

Root cause

A customer's staging integration sent an unusually high volume of rapid, repeated requests to the platform, which overloaded two web servers in the affected region and caused them to hold connections open far longer than normal. Those held connections exhausted a connection-capacity limit on our shared network routing layer, which had not been configured to match its intended capacity — preventing it from opening new connections for any customer routed through it, including customers whose own servers were unaffected.

Resolution

Our engineering and infrastructure teams identified the two overloaded servers and monitored the incident as the source integration traffic tapered off; held connections drained and service was fully restored by 21:37 UTC (4:37 PM CDT) on July 17, 2026. A shorter recurrence on July 20, 2026 (16:30–16:52 UTC) was automatically contained by an existing safeguard that restarted the affected application processes. To prevent further impact, we revoked the credential associated with the source integration on July 21, 2026, held a technical review with the customer's integration team on July 29, 2026, and issued written recommendations on July 30, 2026. We also corrected the connection-capacity configuration on our network routing layer for the affected region, completed August 4, 2026.

Corrective actions

1. Revoked the credential associated with the integration responsible for the excessive request volume, to eliminate the source of the load. (Completed July 21, 2026)
2. Corrected the connection-capacity configuration on our network routing layer for the affected region to align it with its intended capacity. (Completed August 4, 2026)
3. Conducted a technical review with the customer's integration team and issued written recommendations for optimizing their integration's request patterns. (Completed July 30, 2026)
4. The customer will implement the agreed integration changes — including reducing the size of each request, adopting a more efficient method for retrieving records, and adding delays between retries — before the integration credential is reinstated. (Target date: to be confirmed)