nginx "no live upstreams" error (tested)

no live upstreams while connecting to upstream is an nginx error log line. It means every server in an upstream block has been marked temporarily unavailable (by max_fails/fail_timeout) and nginx has none left to try. nginx is not installed on this box, so we reimplemented its documented passive-health-check algorithm in Node against two real backend servers and measured what actually happens when a pool of backends runs out.

1. What this error means

nginx's upstream module tracks failures per server. If a server fails max_fails times within fail_timeout, nginx marks it "down" and stops sending it traffic for fail_timeout seconds. This is a passive health check: nginx only learns about failures from real client requests, it does not probe backends on its own (that needs the paid ngx_http_upstream_hc_module or a module like nginx_upstream_check_module).

When every server in the block is currently marked down, there is nothing left to route to. nginx logs no live upstreams while connecting to upstream and returns 502 immediately, without attempting any connection. This is a specific sub-case of 502 Bad Gateway (one or more backends failed) and 503 Service Unavailable (the service as a whole is unavailable): the distinct angle here is all backends are down at once, not just one, and the response is instant because nginx already knows the whole pool is dead — it doesn't even try to connect.

2. Why it was not reproduced with real nginx

which nginx returned nothing on this box: nginx is not installed, and there is no local package cache or container runtime available to install it within a reasonable budget. This specific error string is generated by nginx's own internal upstream-health bookkeeping (in ngx_http_upstream_round_robin.c), so it cannot be produced by any tool other than nginx itself.

What we measured instead is the actual, reproducible mechanism nginx documents: counting failures per backend, marking a backend down for a fixed window once its failure count crosses a threshold, and short-circuiting new requests once the whole pool is down. We built this in about 90 lines of Node (no listening port of its own — it is driven by in-process calls so it never touches any port outside 8990/8991) and pointed it at two real backend HTTP servers.

3. Test setup

The proxy logic: on each request, filter the upstream list to servers whose "down until" timestamp has passed. If that list is empty, log and return 502 immediately. Otherwise try a live server; on connect failure, increment its fail count and, once it crosses max_fails, set its "down until" to now + fail_timeout.

function handleRequest() {
  const live = upstream.filter(s => s.downUntil <= Date.now());
  if (live.length === 0) {
    log('no live upstreams while connecting to upstream, client: ..., request: "GET /"');
    return { status: 502, body: 'no live upstreams' };   // no connect attempted
  }
  // ...try a live server; on error, increment fails and mark down
}

4. Results: killing backends one at a time

The headline number: once the pool is exhausted, the short-circuit response (step 4) took 0.002–0.006 ms, against 0.68–4.0 ms for a request that actually attempted a TCP connect and got ECONNREFUSED (steps 2–3). That is roughly 100–1000× faster, because the "no live upstreams" path never opens a socket at all — it is a pure in-memory check against the down-until timestamps. This confirms, mechanically, why nginx's own version of this error is instant: it is a short-circuit, not a failed connection.

Step 5 confirms the other documented half of the mechanism: after fail_timeout elapses, the backend is no longer treated as down and gets tried again on the next request (nginx calls this allowing the server back into rotation to see if it has recovered). In our test the backend was still dead, so it failed again and was immediately re-marked down — but the attempt was made, proving the timeout actually expires rather than being permanent.

5. Why it happens suddenly: the threshold cascade

This is the part that surprises people in production. With max_fails=1, our measurements show it takes exactly one failed request per backend to flip a 2-backend pool from 100% capacity to 0%. In step 2, one failure removed 50% of capacity. In step 3, one more failure (on the only remaining "live" server) removed the rest. Two unlucky requests, and every subsequent client gets "no live upstreams" until the 5-second window passes.

Under real load this looks sudden from the outside: error rates go from 0% to 100% in the time it takes two concurrent requests to hit two different ailing backends, not gradually. The usual trigger is backends that are all degraded together — a shared database running out of connections, a deploy that rolled out a crash-on-start bug to every pod, or a dependency timeout that hits every backend near-simultaneously under load. Each backend only needs to fail once (with max_fails=1, the nginx default is max_fails=1 unless set otherwise) to be pulled from rotation, so a brief correlated blip across all backends removes the entire pool before any single backend would be considered "really" broken.

6. How to fix and prevent it (per the nginx docs)

• Raise max_fails, or don't rely only on passive checks. max_fails=1 is unforgiving: one dropped connection (a deploy restart, a brief GC pause) marks a backend down for the full fail_timeout. A higher value tolerates noise; combine with active health checks (nginx plus or third-party modules) that probe backends independently of client traffic.
• Keep fail_timeout short relative to your deploy/restart time, but not so short backends flap. fail_timeout also controls how long a server is excluded, so very large values keep a recovered backend out of rotation needlessly.
• Always have more than 2 backends if you can. With only 2, as this test shows, two failures (one per backend) empties the whole pool. With 5, the same max_fails=1 policy needs 5 near-simultaneous failures.
• Stagger restarts/deploys so you never take down all backends' listeners at once (rolling restarts, one at a time, with a health-check gate before moving to the next).
• Add a backup server. nginx supports server ... backup; entries that only receive traffic when every primary server is down — a deliberate last resort instead of a hard 502.

For the general all-502 and all-503 cases (not specifically an exhausted upstream pool), see 502 Bad Gateway and 503 Service Unavailable.

Written with AI assistance. Measured on 2026-10-02 with Node.js 24.21.0 on the test box. nginx is not installed on this box (`which nginx` returns nothing), so we could not reproduce nginx's own log line directly. Instead we built a ~90-line Node reverse-proxy that reimplements nginx's documented passive-health-check algorithm (max_fails / fail_timeout) against two real backend HTTP servers on 127.0.0.1:8990 and 127.0.0.1:8991, and measured its real behaviour. The exact nginx log wording is quoted from the nginx source/docs and labelled as such, not measured. PlainHeaders is our extension; Node.js is free.