Our scheduled job died at exactly its 15-minute systemd TimeoutStartSec because one worker thread was stuck inside httpcore's synchronous SOCKS5 handshake. That handshake reads from the socket with no timeout, so the httpx.Timeout we configured never applied. One stalled residential-proxy hop blocked as_completed until systemd killed the whole run.

The fix was not a better timeout setting. It was a hard wall-clock watchdog around every proxied request, plus not sending requests through proxies when they could never succeed. Below is the full postmortem: the symptom, how we diagnosed it, why the obvious fixes don't work, and the rule we now apply to every scheduled job we run.

What was the symptom?

After we shipped a proxy fallback chain to a site-triage fetcher, every run of the heavy scheduler lane ended with systemd reporting result: timeout. These weren't occasional slow runs. Each one died at exactly 15:00, the configured TimeoutStartSec, in the six hours between shipping the chain and shipping the fix.

The system is a prospecting platform for a B2B sales team. A oneshot systemd service fires on a timer, triages a batch of company websites with 10 fetch threads, and writes the results to Postgres. Triage has a 420-second soft budget: once it runs out, no new fetches start and in-flight ones are allowed to finish. Clean runs were taking 10 to 13.5 minutes under backlog, so 15 minutes was tight but enough.

The change that started it: when a direct fetch failed with a connect, DNS, TLS or timeout error, or came back as a 403/429/5xx block, we retried it through a datacenter proxy and then through a residential SOCKS5 gateway. It was a sensible resilience feature, and it took down the entire lane.

Why does dying at exactly TimeoutStartSec point to a blocking call?

A slow job dies at different times on different runs, because the amount of work varies. A job that dies at exactly the configured limit every time isn't slow. It's waiting on something that never returns, and the timeout is the only thing ending the run. When the kill time matches the limit, suspect an unguarded blocking call.

The systemd.service documentation spells out the mechanics. For Type=oneshot, systemd waits for the start action to finish, and TimeoutStartSec puts a ceiling on that wait. When the ceiling is hit, the unit is terminated and marked failed. All systemd can tell you is that the process never finished. It can't tell you why.

Our tests never waited on a silent peer. Every fake transport in the suite either returned a result or raised an exception. Production hit the case the tests skipped: a proxy gateway that accepted the TCP connection and then sent nothing back.

How did we find the hung thread?

We attached py-spy to a live run that had passed its soft budget but was still alive. One worker thread was parked in httpcore/_sync/socks_proxy.py, line 97, inside _init_socks5_connection, blocked on stream.read(). The other nine threads were idle. The main thread was waiting in as_completed for that one future.

py-spy matters here because it doesn't need you to have logged the right thing in advance. It samples the stack of a running Python process from outside. Our logs showed the run going quiet after the soft budget. The stack dump showed exactly which frame it was waiting in.

Line 97 is the read that waits for the SOCKS5 server's reply to the CONNECT command. The gateway had accepted our TCP connection and completed authentication, then never answered the CONNECT. Residential proxy pools do this routinely when the exit node they picked drops off.

Why doesn't httpx.Timeout apply to the SOCKS5 handshake?

In httpcore 1.0.9's sync backend, the SOCKS5 handshake calls stream.read(max_bytes=4096) without a timeout argument. SyncStream.read() then calls sock.settimeout(None) before recv(), which puts the socket into blocking mode with no limit. Any timeout httpx configured is removed, and a silent proxy blocks the thread forever.

The connect timeout in httpx.Timeout does reach httpcore, but only the TCP connect to the proxy uses it. The three handshake reads (auth method, username/password, CONNECT reply) don't get it. The async backend passes timeouts correctly, so this only affects httpx.Client, not httpx.AsyncClient.

This is a known upstream bug: httpcore issue #1089, "SyncStream.read() puts socket into infinite blocking mode", reported against httpcore 1.0.9 and httpx 0.28.1, the same versions we run. It is still open, with a fix in an open PR. Until a release ships it, any sync httpx client that uses a SOCKS5 proxy has no real timeout on the handshake.

Why don't the obvious fixes work?

The three obvious fixes all fail for one reason: none of them can stop a thread blocked in a system call. A timeout on as_completed lets the caller stop waiting, but executor shutdown still joins the stuck worker. A soft budget only stops new work starting. Tighter httpx timeouts never reach the handshake read.

The executor point catches people out. The concurrent.futures documentation says ThreadPoolExecutor worker threads are joined before the interpreter can exit. So even if you pass a timeout to as_completed and break out of the with block, the interpreter still waits on the hung worker at shutdown, and systemd still kills the run at 15:00.

We also ruled out switching triage to async. The async SOCKS path handles timeouts correctly. But triage is sync code that runs inside a sync scheduler and writes to the database from the calling thread. Rewriting the transport layer to fix one handshake was a lot of risk to take on to close a single bug, and the watchdog pattern covers every future blocking call too, not just this one.

What does the watchdog fix look like?

Each proxied request now runs in its own daemon thread, and the caller waits on an event with a hard limit of FETCH_TIMEOUT plus 5 seconds. If the limit passes, the caller abandons the thread and records the attempt as failed with the error code proxy_watchdog_timeout. The chain then moves on to the next proxy tier.

# simplified shape of the pattern
def fetch_via_proxy_guarded(url, limit):
    box, done = [], Event()

    def run():
        box.append(fetch_through_proxy(url))   # may block forever
        done.set()

    Thread(target=run, daemon=True).start()
    if not done.wait(limit) or not box:
        return Failed("proxy_watchdog_timeout")   # abandon, don't join
    return box[0]

We chose a daemon thread over a pooled worker specifically because daemon threads never block interpreter exit. The runs are oneshots, so an abandoned thread can only live until the process ends, which caps the leak at one run. If the late result ever lands in box after we've given up, nothing reads it, so it can't overwrite a result we already recorded.

One less obvious detail: we stopped passing the per-host rate limiter into proxied requests. An abandoned thread would hold its per-host slot until the process ended and starve every later fetch to that host. The direct attempt had already been rate-limited for that host, so the proxied retry didn't need its own slot.

Why skip the proxy chain for Cloudflare challenges?

Most of the proxied requests were never going to succeed. A Cloudflare bot challenge checks JavaScript execution and the TLS fingerprint. A datacenter proxy shares our IP reputation, and a residential SOCKS5 exit through httpx still presents an httpx TLS fingerprint. Neither can pass. Each challenged site was spending up to two fetch timeouts of the run's budget for nothing.

We'd learned this the hard way before, while building an on-demand proxy fleet to collect 1.1M records from a bot-protected portal: against fingerprinting bot protection, a raw HTTP client fails regardless of which IP it comes from. A different IP doesn't fix a TLS fingerprint check.

We now treat a response as a challenge if it has cf-mitigated: challenge, or if it has server: cloudflare with a 403, 429 or 503. Those responses skip the chain and go straight to deferral. A Cloudflare 502 is an origin error, not a challenge, so it still goes through the chain. That distinction has its own test case.

This change shortened every run whether or not the watchdog ever fired, and it cut down how often we hit the flaky SOCKS5 path in the first place. Removing calls that can't succeed made the system more reliable than guarding them did. A request through a proxy uses up both run time and proxy capacity, so a request that can't succeed is the first one to cut.

Did we also raise the systemd timeout?

Yes, but only after the watchdog was in, and for a different reason. The worst-case budgets of every job in the heavy lane (triage's 420 seconds plus stragglers, intel, and discovery at up to 480 seconds) add up to about 15 minutes under backlog. We raised the heavy lane to 25 minutes. The fast lane stays at 15.

The order matters. Raising the limit first would have made each hung run waste 25 minutes instead of 15 and fixed nothing. A service timeout should be the sum of the job's own budgets plus some headroom, not a number you raise until the alerts stop. If the timeout is what ends a run, you haven't found the bug yet.

The fix shipped with 24 new test cases. One hangs a fake transport on an event that never fires, to prove the watchdog abandons it. Others check that normal results pass through, that exceptions turn into failed requests, that a watchdog failure moves the chain to the next tier, and that both Cloudflare and non-Cloudflare block patterns are classified correctly. The triage suite now runs 160 tests, all passing.

What rule did we adopt for every scheduled job?

Watchdogs everywhere: no network call inside a scheduled job may rely only on the library's timeout. Every call into a transport we don't control runs under a wall-clock limit enforced by our own code. When a job dies at exactly its TimeoutStartSec, the first suspect is a new unguarded blocking call, not slow work.

This is now written into the operations notes for the system, so the next engineer who sees a run killed at the limit starts with py-spy and the list of transports. A library's timeout setting only covers the code paths the library actually applies it to, and the SOCKS5 handshake in sync httpcore isn't one of them. That gap is why a 15-second fetch timeout turned into a 15-minute outage.