>_C2CZ
Serverless · Self-Hosting · Security · Independent Ops

Cloudflare Workers: "code had hung" Error 1101 fix

Error: The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. is not a timeout and it is not a slow origin. The runtime emits it the moment it can prove that the request's handler will never return a Response — and the most common way to build that situation is a promise that one request created and a different request awaited.

I reproduced both shapes of that mistake in workerd 2026-09-15, the engine behind wrangler dev, on an arm64 host with Node 26.5.1. A module-scope barrier promise that another request resolves is refused in 1.2 ms with an HTTP 500 and two distinct log lines — one of them names the file and line of the resolve call. A cached in-flight fetch() awaited by the next request just hangs: 45 s with no response and, at default log verbosity, no diagnostic at all. In production the first of those reaches the user as Error 1101; the second never answers.

Every number and every quoted log line below comes from those runs on 2026-09-15. If you are here because production is down, jump to the grep list — it is four patterns, and one of them is almost certainly your bug.

TL;DR

  • The cancellation is proof-based, not timer-based. Cloudflare's wording is that it happens when "all the code associated with the request has executed and no events are left in the event loop, but a Response has not been returned". Measured: 1.2–1.6 ms across four runs (1.204, 1.215, 1.258, 1.647 ms). Nothing here is waiting 30 s for an origin.
  • A promise resolved by another request gives you a log line that names the culprit. Warning: A promise was resolved or rejected from a different request context ... followed by at Object.fetch (worker.js:26:7) — line 26 was the resolveReady("go") call, inside the request that resolved it.
  • A cached in-flight fetch() is the silent version of the same bug. The awaiting request never gets an answer (measured ≥45 s) and workerd says nothing about it at default verbosity. That is the one that looks like a hung origin in your dashboard.
  • Promises belong to the request that created them. Plain data does not. Cache the resolved value, report state instead of awaiting another invocation's promise, and reach for Durable Objects or Workers KV when you need real cross-request coordination.
  • The compatibility flag the warning suggests removes the warning, not the failure. With no_handle_cross_request_promise_resolution set, the awaiting request still returned 500 in 1.2 ms, and the resolving side then threw Cannot perform I/O on behalf of a different request.
  • You can reproduce all of it with no Cloudflare account. workerd ships these exact strings; the lab below is one Cap'n Proto config and one file you swap between broken and fixed.

Vitest Workers: one workerd per test file, 8.08s to 5.08s

A Workers Vitest suite boots one workerd instance per test file, and that boot is invisible in Vitest's own summary line — so a suite whose assertions take milliseconds still pays half a second of runtime startup per file. Counting pgrep -x workerd during a serialised run of 14 test files returns exactly 14. Move the files that do not touch a Workers API out of the Workers pool and the same suite spawns 6, with wall clock down from 8.08 s to 5.08 s at default settings.

I measured this on a 4-vCPU arm64 box (Node 26.5.1, Vitest 4.1.11, @cloudflare/vitest-plugin 1.1.9, workerd 2026-09-11) with a suite of 6 integration files against a Hono Worker plus 8 unit files over its plain-TypeScript helpers — the mixed layout most Workers repositories drift into. Every figure below comes from that run, three runs per cell, medians quoted. If you still depend on @cloudflare/vitest-pool-workers, the older package name, the accounting is identical; it exposes the same pool options and Cloudflare's documentation now points at the @cloudflare/vitest-plugin Vite plugin.

TL;DR

  • One test file, one workerd process. A serialised run of 14 files spawns 14 runtimes. After the split below it spawns 6 — one per integration file, none for the unit files.
  • The per-file cost is not in the summary buckets. A unit file that imports no Cloudflare API at all takes 728 ms of wall clock inside the pool and 220 ms in a plain Node project. About 650 ms of the 728 ms is neither transform, import, tests nor environment — the line does not show it, and 14 files pay it 14 times.
  • Do not trust the import bucket as your signal. The same integration file reports import 1.28 s / tests 13 ms with the deprecated SELF binding and import 37 ms / tests 1.24 s with the recommended exports.default.fetch() — identical 1.96 s wall clock, identical work, different buckets. Wall clock and process count are the measurements that survive.
  • SELF and env from cloudflare:test are deprecated in @cloudflare/vitest-plugin 1.1.9; the shipped types point at import { env, exports } from 'cloudflare:workers' and exports.default.fetch(). Both work — I ran them side by side.
  • The fix is a Vitest project split, not a pool option. Only integration files stay in the Workers pool. Wall clock at default settings: 8.08 s → 5.08 s (−37.1%), runtimes 14 → 6.
  • Raising maxWorkers buys contention, not throughput. 1 → 8 cuts wall clock from 12.74 s to 7.10 s while multiplying the aggregate test-phase work by 4.06× (4.22 s → 17.14 s) on four cores.
  • --isolate=false is the trap. It collapses 14 runtimes into 1 and 12.74 s into 2.42 s — and it breaks test isolation. My four-file probe: 4/4 pass with isolation, 2 failed / 2 passed without it (expected 1 to be +0, expected [ Array(1) ] to deeply equal []).

Cloudflare Workers: "This ReadableStream is disturbed" Fix

TypeError: This ReadableStream is disturbed (has already been read from), and cannot be used as a body. means one specific thing: your Worker already consumed the request body, and a Fetch body is a single-use stream. Nothing is broken in the runtime, the binding or the deploy — a line above the one that threw read the bytes, and every later read is refused by the spec.

The reason this error wastes afternoons is that the same mistake prints a different message depending on which call touches the used body. Reading it twice gives you Body has already been used; cloning too late gives you currently locked to a reader; rebuilding or forwarding the request gives you the disturbed string above; and on Node instead of workerd you get a fifth wording entirely. I reproduced every one of those triggers against workerd 2026-09-11, Node 26.5.1 and Hono 4.13.7 while writing this, so the map below is measured, not paraphrased. After that comes the handler I now ship for signed webhooks: one read, from the stream, into a buffer the rest of the request lifecycle uses.

TL;DR

  • A Fetch body is a one-shot stream. request.text(), .json(), .formData(), .arrayBuffer() and a raw request.body.getReader() all consume it. The second consumer throws.
  • workerd words the double-read case plainly: TypeError: Body has already been used. It can only be used once. Use tee() first if you need to read it twice. This is the message you get from two accessors on one request.
  • The disturbed string in this article's title is the rebuild/forward path: new Request(url, alreadyReadRequest). That is the retry-with-failover and proxy pattern, and it fails at construction time.
  • Forwarding a used request to a service binding throws a third message: TypeError: Cannot reconstruct a Request with a used body.
  • clone() is only valid before the first read. Afterwards it throws TypeError: This ReadableStream is currently locked to a reader,. Clone first, then read.
  • Node and undici say TypeError: Body is unusable: Body has already been read for the same mistakes — worth knowing when the same handler logic runs in a Worker and in a Next.js route.
  • Hono caches parsed bodies, so c.req.json() twice is safe — but c.req.raw bypasses that cache, and mixing the two reproduces the error exactly. Put Hono's own bodyLimit middleware in front and read through a cached accessor, and neither the error nor an unbounded buffer is reachable.
  • The fix is architectural, not defensive: read the body once into bytes with a hard size cap, verify the signature over those bytes, then parse from the buffer. Full Worker and test harness below, executed on workerd and on Node.

System limit for number of file watchers reached in Docker

ENOSPC: System limit for number of file watchers reached does not mean your disk is full — it means the kernel refused a new inotify watch because your user ID has already spent its entire watch budget. The word that misleads everyone is ENOSPC: on a filesystem it means "no space left on device", but inotify_add_watch(2) reuses the same errno for "the user limit on the total number of inotify watches was reached". Nothing is broken, nothing is leaking, and there is no file to delete.

I meet this in self-hosted stacks rather than on laptops: a hot-reloading dev container that stops noticing edits after a few hours, a Syncthing or filebeat sidecar that goes quiet, a tsc --watch inside a container that works on Monday and throws on Tuesday. The pattern is always the same. The limit is a property of the host kernel and is accounted per real user ID, so every container and every host process running as the same uid is spending from one shared pot. This article is the version of the fix I use on client hosts: first work out which of the two inotify budgets ran out, then raise it on the host where it actually lives. Everything quoted below was executed on a Linux 6.17 host as uid 10000, or taken verbatim from the kernel and Docker documentation.

TL;DR

  • ENOSPC here is a watch budget, not disk space. inotify_add_watch(2) returns it when the per-user watch count is exhausted; your filesystem is fine.
  • There are two budgets, with two different errnos. Watches (fs.inotify.max_user_watches) fail with ENOSPC; instances (fs.inotify.max_user_instances) fail with EMFILE. I reproduced the second one on demand: 124 instances opened, then errno 24 EMFILE (Too many open files) against a limit of 128.
  • Read the real numbers before you change anything: /proc/sys/fs/inotify/max_user_watches, max_user_instances, max_queued_events. The lab host runs 188727 / 128 / 16384.
  • A container cannot raise its own limit. The budgets are per real user ID in the kernel, and Docker only accepts namespaced sysctls per container — its own documentation says it "does not support changing sysctls inside a container that also modify the host system". The fix happens on the host, or in the VM that runs the engine.
  • Not every missed event is a limit. A watch on a file dies when an editor saves atomically: the kernel sends IN_ATTRIB then IN_IGNORED, and the watcher is deaf forever. Watch the directory instead, or use the library's atomic option.
  • Docker Desktop is a different bug with the same symptom. On macOS and Windows the events may never cross the host/container boundary at all — and Docker's own known-issues page states outright that inotify does not work under QEMU emulation.

self-signed certificate in certificate chain: Fix curl 60

curl: (60) SSL certificate problem: self-signed certificate in certificate chain means the chain your client received ends in a self-signed root that is not in its trust store. Nothing is broken on the server side, and the certificate you fetched is almost certainly valid — what is missing is the one certificate that signs it, usually an interception proxy, a VPN gateway, or an internal PKI root. Disabling verification is not the fix; making the client trust that specific root is.

I run into this constantly on engagement workstations and build agents: the same request that works from a laptop at home dies in a corporate network, or the reverse — a service that talks happily to the internal API chokes on github.com the moment someone points an environment variable at the corporate CA. The failure is never one bug. It is four different trust stores silently disagreeing, and the error message only names the last client that noticed. Everything below was reproduced on a lab origin with a private root CA; the command outputs are the real ones.

TL;DR

  • Error 19 (this article) is a missing root CA — the leaf is signed by a self-signed CA your client has never seen. That is what TLS interception and internal PKI look like. Error 18 is a self-signed leaf; error 20/21 is a server that forgot to send its intermediate. Different fixes, same exit code 60.
  • Identify before fixing: openssl s_client -connect host:443 -servername host prints verify error:num=19:self-signed certificate in certificate chain. The certificate you must trust is the self-signed one — the last entry when the root is sent, or the issuer named in the last entry when the chain stops at an intermediate.
  • Install the CA once, system-wide (Debian/Ubuntu: /usr/local/share/ca-certificates/*.crt + sudo update-ca-certificates). That covers curl, OpenSSL, git and anything else on the OpenSSL default paths.
  • It does not cover everything. Python's requests verifies against certifi's bundle, not your OS store, and SSL_CERT_FILE, SSL_CERT_DIR and REQUESTS_CA_BUNDLE replace the bundle they point at. Set one of them to a private CA alone and you break every public site for that process.
  • Node and git are the easy ones: NODE_EXTRA_CA_CERTS is additive (verified), and http.sslCAInfo / GIT_SSL_CAINFO accepts a private bundle.
  • Never "fix" this with -k, --insecure or NODE_TLS_REJECT_UNAUTHORIZED=0. If you cannot account for the root you are being asked to trust, you are not looking at a configuration problem, you are looking at a man in the middle.

Cloudflare Error 1042: Worker Fetch to Same Zone Fails

Your Worker's fetch() call to a hostname on its own Cloudflare zone dies with Error 1042 — Worker tried to fetch from another Worker on the same zone, while the identical code passes in wrangler dev. This is not a runtime bug and not a DNS problem: it is a routing decision. A subrequest from a Worker back into its own zone is ambiguous — the runtime cannot tell whether it should be trusted and sent straight to the origin, or untrusted and pushed through Cloudflare's front door with every Worker, rule and WAF check re-applied — so it refuses the ambiguous case outright unless you opt in.

I shipped exactly this on a split Worker setup: an app Worker on app.example.com calling an auth Worker on auth.example.com, same zone, through its public hostname. Preview was green, local was green, and production served 1042 error pages to every request that needed a token. The routing rule that broke it takes one config change to satisfy — but only if you know which of the three fixes applies to your case.

TL;DR

  • Error 1042 means the runtime refused a same-zone Worker-to-Worker subrequest. Cloudflare's Workers errors documentation defines it as "Worker tried to fetch from another Worker on the same zone, which is only supported when the global_fetch_strictly_public compatibility flag is used". It is one of the error pages a Worker generates when it cannot return a response; like the other 1xxx codes it appears in the HTML body of the response, not in the HTTP status.
  • Why the rule exists: when a Worker subrequests its own zone, Cloudflare has no way to know whether that call is internal (trusted, should go to the origin) or external (untrusted, should re-enter the front door). The default is the origin — Workers mapped to that URL are skipped. The runtime now fails fast with 1042 instead of silently bypassing your own code.
  • Fix order: a Service binding (RPC or HTTP) is the documented path and the one Cloudflare recommends; global_fetch_strictly_public is the escape hatch when you specifically want the public path; a dedicated non-apex hostname is the workaround for zone-internal endpoints such as /cdn-cgi/media/.
  • Local development lies to you. wrangler dev runs a local runtime and does not apply same-zone routing, which is why this class of bug only appears after wrangler deploy. There is an open-then-closed issue about exactly that (workers-sdk #11215).
  • It is not the same failure as Error 1101 (your code threw), Error 1102 (resource limit), Error 1016 (edge DNS cannot resolve the target) or Error 1020 (a policy blocked the request). 1042 happens before any of them: the subrequest never reaches a handler.

device descriptor read/64, error -71: Raspberry Pi USB fix

When a USB device keeps dropping off a Raspberry Pi, the kernel log fills with device descriptor read/64, error -71 and the device never finishes enumerating. This is not a driver problem, a filesystem problem, or a sign that the board is dying: the USB bus itself is failing the very first handshake with the device, and on a Pi the root cause is almost always electrical.

I have chased this exact log line on two different rigs: a Pi 4 kiosk whose 4G modem vanished every few hours, and a Pi 3 that refused to talk to a printer controller board hanging off a powered hub. Same error, different physics. The log shows a storm — reset, descriptor read, fail, reset again — and most forum advice jumps straight to kernel parameters. That is backwards. The fix order that actually works is: power, cable, hub, then autosuspend. Kernel tweaks come last, and usually are not needed at all.

TL;DR

  • -71 is -EPROTO (protocol error). During enumeration the kernel sends a GET_DESCRIPTOR control transfer, and the device answers with data that fails validation, or the transfer dies mid-flight. The kernel logs the failure and retries with port resets.
  • The repeating lines are normal kernel behavior, not a crash. Each plug event ends with device not accepting address N, error -71 and unable to enumerate USB device. The port stays dead until you unplug and replug.
  • On a Pi, power is the first suspect. Run vcgencmd get_throttled: bit 0 means undervoltage right now, bit 16 means it has happened since boot. An adequate supply or a powered hub resolves most cases.
  • If it only fails after idle, suspect USB autosuspend. Disable it with usbcore.autosuspend=-1 on the kernel command line — a two-line change, no driver rebuild.
  • Isolate before you configure. Try a known-good device on the same port, then the failing device on another machine. Ten minutes of swapping hardware beats a day of kernel options.

💡 The same workflow applies to other kernel-level device failures on the Pi. We have already torn down mmc0: error -110 whilst initialising SD card (an SD-card controller timeout — same -110 ETIMEDOUT family you will see when a USB device does not answer at all) and Klipper "MCU 'mcu' shutdown: ADC out of range" (a controller board that talks to the Pi over USB). Capture the exact log line first, then fix the physical layer.

Cloudflare Error 1020 Access Denied: Fix

Cloudflare Error 1020 "Access denied" means a firewall rule evaluated your request and blocked it — nothing on the origin server is broken, and Cloudflare is working exactly as configured. The error page you land on is Cloudflare's branded block page, not a response from the website's server, and the only identifier that matters on it is the Ray ID at the bottom.

I have debugged this error from both sides of the glass: as a visitor locked out of a site I legitimately needed, and as the operator who finds the matching rule in Security Events. The two fixes could not be more different, which is why most advice online is useless — it assumes you are the site owner. This post covers the exact triage for each role, with the Cloudflare dashboard paths and a GraphQL query you can run today to pull the offending event out of the analytics API.

TL;DR

  • Error 1020 is a deliberate block, not an outage. A Cloudflare firewall rule with a Block action matched the request at the edge and answered with a 403 and the branded "Access denied" page. The origin never saw the request.
  • As a visitor, you cannot fix it. Retrying does nothing (you are still you), and hammering refresh can make adjacent automated rules treat you as a bot. Capture the Ray ID and the time, then contact the site owner through any channel that is not the blocked page.
  • As the site owner, the Ray ID is your lookup key. Open Security Events (Security → Analytics → Events), paste the Ray ID or the visitor's client IP, and the log shows exactly which rule matched and what action it took.
  • Free and Pro plans only retain firewall events for about 24 hours, so trace 1020 reports the same day you get them — or the evidence is gone. Business gets 3 days, Enterprise 30.
  • The common root cause is a stale rule. The block is usually years old and over-broad — a country or ASN filter written when the threat model was different. The durable fix is narrowing the rule, not whitelisting one annoyed customer.

⚠️ Do not confuse 1020 with Cloudflare errors that mean the edge itself failed. Error 1016 (origin DNS error) and Error 1101 (worker threw an exception) are infrastructure faults; 1020 is a policy decision. Different code, different fix, different inbox.

Docker 'request canceled while waiting for connection': Fix

docker pull and docker login fail with net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) when the Docker daemon cannot complete a connection to Docker Hub. The error text is the same whether the real fault is a missing proxy, broken IPv6 routing, DNS flapping, or an egress firewall — so the fix is never "restart Docker" first, it is finding which layer dropped the connection.

The main Stack Overflow thread on this error has been open since December 2020 with 10,000+ views and zero accepted answers. Most advice in the wild tells you to add 8.8.8.8 to daemon.json — I have seen that fix fail repeatedly, because the daemon never consults that setting for its own registry connections. This post walks the diagnosis in the order that actually isolates the cause, with commands I verified against a live registry path on 2026-09-07.

TL;DR

  • The message is a dial-phase timeout. "request canceled while waiting for connection" means the TCP connection never finished; "(Client.Timeout exceeded while awaiting headers)" is the client-side timeout wrapper. This is not a TLS error, not a credentials error, and not a disk error.
  • Two endpoints must be reachable, not one: registry-1.docker.io:443 (manifest and blob traffic) and auth.docker.io:443 (the bearer-token service Docker Hub uses on every authenticated pull).
  • Probe with curl before touching any config. An HTTP 401 from https://registry-1.docker.io/v2/ means the network path is healthy — 401 is Docker Hub's correct answer to an unauthenticated request.
  • daemon.json's "dns" key is a container setting. It does not change where the daemon resolves registry-1.docker.io. Fixing DNS at the host layer is what actually works.
  • Fix at the layer that connects: daemon proxy (systemd drop-in or daemon.json proxies), host IPv6 routing, host DNS, or a registry mirror. Order matters — start with the curl probes below.

Cloudflare Error 1016 Origin DNS Error: Fix in Workers

Your browser is showing Cloudflare's Error 1016 page — "Origin DNS error" — and the request died at the edge, not in your application. Error 1016 means Cloudflare could not resolve the IP address of the origin it was told to connect to: a DNS problem sitting in front of your code. On Cloudflare Workers it almost always means one of two things: a Worker fetch() subrequest hit a hostname the edge resolver cannot resolve, or the DNS record behind your route or custom domain is missing, stale, or dangling.

This one cost me a morning on a Worker that calls an internal API on a subdomain of the same zone. Locally, wrangler dev answered fine. At the edge, every request came back as a Cloudflare error page with status 530. I redeployed twice before I stopped blaming the code and looked at what Cloudflare's resolver could actually see — which is not the same DNS your laptop sees.

TL;DR

  • Error 1016 = origin DNS failure. Cloudflare cannot resolve the origin server's IP address. The visitor-facing page says Error 1016 Origin DNS error; the underlying HTTP status is Cloudflare's internal 530.
  • Two distinct Worker causes: (1) a fetch() subrequest to a hostname in a Partial (CNAME) setup zone that has no DNS record inside Cloudflare; (2) the origin record behind your own hostname is missing (no A record), points at a dead IP, or is a CNAME to a target that no longer resolves.
  • Subrequest failures split into two modes. A Cloudflare-routed destination that fails edge DNS comes back as a Response with status 530 (1016) — inspect response.status, because a try/catch around fetch() alone misses it. A plain network or DNS failure (an unresolvable hostname that is not Cloudflare-routed, a refused connection) makes fetch() reject with a TypeError — that one belongs in the catch path.
  • Fix order: identify the failing hostname → prove whether it resolves (dig/getent) → add or repair the DNS record in the Cloudflare dashboard → replay the request. No code change fixes a missing record.
  • Related reading: this is DNS-level, so it sits next to Error 1101 (unhandled exception) and Error 1102 (resource limits) as "the Worker runs, but the edge refuses" — and is a cousin of Cloudflare 524, where the origin answers too slowly instead of not resolving at all.

C2CZ

Hyper-specific engineering guides: Cloudflare Workers, Turso, self-hosting, network security, offensive security, independent operations.