Status: Resolved
Duration of customer-visible impact: Friday, September 4, 2026 (initial, sparse reports) – Tuesday, September 8, 2026, 6:00 PM (resolved)
Severity: Intermittent multi-second delays loading instance data in the Management Console. No data loss, no downtime, no security impact.
Summary
Starting Friday, September 4, 2026, a small number of users of the UMH Management Console reported their instance list and related pages occasionally taking anywhere from 1s to nearly 30s to load. The issue escalated sharply on the morning of Monday, September 7, and stayed disruptive until it was resolved on Tuesday evening. The Management Console backend itself was never the source of the delay. Every affected request was handled by our server in a few milliseconds. The delay was occurring on the network path between our CDN edge (Cloudflare) and our hosting provider's origin servers (fly.io), outside of our application code.
Two smaller, real performance issues were also found and fixed along the way. Neither of these was the primary cause of the reported slowdowns, but both were contributing to a less responsive Console overall and are now resolved.
The root network issue was resolved on Tuesday, September 8, 2026, around 6:00 PM, by changing how traffic is routed between our CDN and our origin infrastructure.
Impact
- Users saw the instance list, and other pages that depend on it, hang for one to thirty seconds before loading, on an unpredictable basis.
- The latency between the Management Console and the backend was permanently high.
- The problem was more visible in production than in our staging environment, which led us to initially suspect it was load-related. That turned out to not be the case -> see "What we got wrong" below.
- No requests failed outright in a way that lost data; the console recovered on retry or once the slow request eventually completed. A small number of requests during testing did surface as connection errors at the CDN layer.
Timeline
- Friday, Sept 4, 2026 (evening) — First, sparse reports of slow loading in the Management Console. Investigation opens. First contributing issue fixed and deployed the same day: an internal service-to-service HTTP client was reusing far fewer connections than it should have, forcing unnecessary new connections under load. This had been adding a chronic ~550–650ms to a portion of requests. Fixed by correctly configuring connection pooling.
- Monday, Sept 7, 2026 (morning) — Reports escalate sharply; the issue becomes acutely disruptive. Investigation intensifies to find the primary cause.
- Sept 7–8, 2026 — Backend response times were confirmed fast on every single request, including the slowest ones, which ruled out the application and database as the source of delay. Systematic testing isolated the delay to the network segment between our CDN edge and our origin servers.
- Tuesday, Sept 8, 2026 (~6:00 PM) — Root cause confirmed and resolved: we enabled Argo Smart Routing on our CDN and configured a dedicated origin route pointed at our primary hosting region instead of a general global endpoint. Stalls stopped.
What happened
The Management Console's backend was measured at ~10ms to handle the affected request, every time, including on the run that took the users up to ~30s end to end. That ruled out our application code, our database, and our business logic from the very first solid measurement. This meant the delay was happening somewhere between the user's browser and our server actually starting to process the request.
We instrumented the browser side to split each request's total time into the portion our server reported handling versus everything else, and confirmed the "everything else" portion was where nearly all of the delay lived on every stalled request.
We then tested each segment of the path independently: our CDN edge alone, our CDN to origin, our load balancer to our backend directly, and our backend in isolation, bypassing the CDN entirely. Every path that excluded the CDN-to-origin segment was clean. Every path that included it showed a high rate of spikes and, in a few cases, the CDN returning an error because it never got a valid response from our origin at all. That isolated the fault to packet loss on the network segment between our CDN's edge and our origin servers.
What we got wrong along the way
Early on, the problem looked production-only, which pointed us toward a load or capacity explanation and cost us time. It didn't hold up: our staging environment, which is provisioned with tighter resource limits than production, showed the exact same failure pattern in one of our test runs. The "production only" appearance was more likely because our existing latency tooling measured from the server side, which is blind to a problem that happens before the server sees the request, and because our synthetic checks reconnected on every sample instead of reusing connections the way a real browser session does. Once we started measuring from a real client with a persistent connection, the problem was visible everywhere we looked for it, not just in production.
Contributing issues (fixed, but not the primary cause)
Connection pool exhaustion. An internal service that proxies requests between our components was using Go's default outbound connection pool size, which is far too small for our production traffic. This forced a fresh connection setup on most requests instead of reusing an existing one, adding a steady ~550–650ms of latency to a portion of traffic. Fixed by correctly sizing the connection pool on September 4.
Stale concurrency limits. Request-concurrency limits on our backend infrastructure were not dialed in perfectly. Production traffic was hitting a limit, causing requests to queue at the infrastructure layer before ever reaching our application. Limits were raised to reflect our current setup and load.
Resolution
The underlying network issue was resolved by two changes on the CDN side, made together around 6:00 PM on Tuesday, September 8, 2026:
- Configuring a dedicated origin route pointed directly at our primary hosting region, rather than a general-purpose global endpoint. See the Technical details section below for more info.
- Enabling Argo Smart Routing, which routes traffic over less congested network paths instead of the default path.
Since this change, we have seen no recurrence of the retransmission pattern or the associated stalls.
Technical details
Both Cloudflare and Fly.io use anycast networks, so finding a specific route between a users end device and our backend servers is not always a trivial task. Before this incident, we used a dedicated IPv4 address for routing between Cloudflare and Fly.io, but this IP address was scoped as global. This meant, that anycast was still used under the hood.
We now scoped the IP address to the region of our primary datacenters and voila - No more routing issues.
Follow-up work
- Reduce a small amount of remaining unnecessary database work on one of our endpoints, identified during this investigation as slower than it needs to be.
- Improve the efficiency of how our backend checks for new messages to relay to the console, to reduce load on a shared internal cache as our fleet of connected instances grows.
- Build permanent client-side latency monitoring so a problem like this is caught automatically rather than relying on user reports.
We're sorry for the disruption this caused. If you're still seeing slow loads in the Console after September 8, please reach out to us so we can investigate.
