Web App, Windy and MCP Unavailable

Background

Windmill generates Windy reports in the background to prepare content for our users. On October 12, 2026, a large number of these reports started at the same time.

Our services connect to the database through a connection proxy. The proxy usually shares a small number of database connections between many requests. But a problem in how our services use their database connections stopped the proxy from sharing them. Each request held its own...

Web app degraded due to increased load. Additional resources have come online.

Background

Early this week, Windmill rolled out enhanced monitoring for the core infrastructure which processes all requests coming into all our services (Windy, web app, MCP, and integrations).

We noticed a small number (>0.01%) of requests were failing due to a networking misconfiguration in how our services talk to each other. Although there was no customer impact since these failing requests get retried, we proactively rolled out a fix the evening of 9/30/2026.

Windmill runs its serv...

Services have been scaled up and are now stable.

We are continuing to monitor this but the syncing system is healthy now and is working as normal

The issue has been resolved. Approximately 2% of users were impacted and we have emailed everyone affected

Excess load caused the degradation, more resources came online. This is resolved.

This has been resolved

Additional resources have come online. This is now resolved.

We are continuing to monitor this but the syncing system is healthy now and is working as normal

A fix has been released. Meeting syncing will gradually resolve over the next day.