Host key verification failed: six causes, one test

By Mara Voss · C2CZ · 28 October 2026 · Last verified: 2026-10-28

Mara Voss covers Docker, system limits and reverse proxies at C2CZ. Author page · All authors.

Host key verification failed. is the last line of an ssh run, not the first clue. OpenSSH prints it whenever it cannot confirm the server's identity against a key it trusts, and that one sentence covers at least six unrelated faults: an empty known_hosts in a fresh container, a cron job with no terminal to answer the prompt, a config block that never applies because of its position, and a key file that silently refuses the write and reports success anyway. This post reproduces each case against a real server, quotes the diagnostic line that separates them, and ends with a provisioning step that survives a redeploy.

Prerequisites: an OpenSSH client, a reachable host and port, and write access to your own known_hosts. These runs used OpenSSH 10.0p2 on Debian 13, aarch64, talking to a server that presents a freshly generated ED25519 host key on port 2222 — option names and defaults differ on older releases, so re-run the one-liners below on your own client before quoting them to a colleague.

One disclosure about the rig: this host has no sshd, so the server end is AsyncSSH 2.24.1 with a key generated by ssh-keygen. The client is genuine OpenSSH, and every string quoted below is emitted by that client — which is the side the error lives on. Server-side policy errors would look different.

TL;DR — the verdict is generic, the line above it is not

• The failure is always Host key verification failed. with exit status 255. The diagnosis sits one line higher and only appears with -v: No ED25519 host key is known for [127.0.0.1]:2222 and you have requested strict checking.
• StrictHostKeyChecking defaults to ask, and ask needs a terminal. Under BatchMode=yes — cron, CI, every Ansible task — the prompt cannot happen, so the default behaves like yes and fails.
• ssh_config is first-match-wins. A Host * block placed above Host lab-* makes the specific block dead code, with no warning anywhere.
• UserKnownHostsFile=/dev/null plus StrictHostKeyChecking=yes does not skip the check, it guarantees the failure. /dev/null only makes sense with accept-new.
• Two widely repeated explanations are false on a current client: an entry stored without brackets does match a non-standard port, and an unset HOME does not break the write.
• accept-new against an unwritable file prints Failed to add the host to the list of known hosts and still exits 0. The run passes, nothing is stored, the next run fails.

tar: file changed as we read it (cron backup fix)

By Noah Lindqvist · C2CZ · 21 October 2026 · Last verified: 2026-10-21

Noah Lindqvist covers hardware, processes and Unix internals at C2CZ. Author page · All authors.

A nightly tar job exits 1 and prints tar: site/access.log: file changed as we read it even though nothing is broken. The archive is usually valid and the message informational — but how your cron wrapper reads that exit code decides whether you get a silent gap in your backups or an alert you will eventually mute. This post measures what the message means on GNU tar 1.35, what each "silencer" flag does to the exit status, and which fix removes the race instead of hiding it.

Prerequisites: a Linux host with GNU tar (these flags are GNU-specific; BusyBox tar and bsdtar differ), a cron or systemd timer you can edit, and shell access to the directory being archived. Everything quoted ran on Linux 6.17.0-1019-oracle, aarch64, GNU tar 1.35 — the tar transcript on 20 October 2026, the audit and wrapper runs on 21 October.

TL;DR — exit 1 means "one file moved", not "backup failed"

• file changed as we read it is a warning: tar recorded the size when it opened the file, copied that many bytes, then re-stat'ed and found the size or mtime had moved. Exit status 1.
• Exit 2 is the fatal one (“Error is not recoverable”): bad archive path, unreadable directory, I/O error.
• --warning=no-file-changed hides the text and leaves exit 1 — evidence removed, monitoring still paging. --ignore-failed-read does the reverse: exit 0, warning kept.
• tar -cf - … | gzip without pipefail returns 0 while tar returned 1: rc=0 without set -o pipefail, rc=1 with it.
• copytruncate produces a different message — File shrank by N bytes; padding with zeros, which --warning=no-file-changed does not silence.
• The archive is not corrupt: it holds exactly the bytes tar saw, short only of the tail written after fstat() — 46 bytes in the run below.

JWT InvalidSignatureError: Signature verification failed

By Kai Tanaka · C2CZ · 17 October 2026 · Last verified: 2026-10-17

Kai Tanaka covers SSH, TLS and applied cryptography at C2CZ. Author page · All authors.

Your token looks right, the payload decodes, and PyJWT still answers InvalidSignatureError: Signature verification failed — with no field, no offset, and no hint about which of six different mistakes you made. That single message is raised for a wrong secret, a secret with a trailing newline, a base64-encoded secret on only one side, a rotated RSA key, an empty signature segment, and a genuinely forged token. They are indistinguishable from the outside, which is why the bug survives three rounds of "but the secret is the same in both services".

This is the diagnostic path I use on self-hosted stacks: reproduce each cause on one machine, learn which exceptions are not this error, then name the cause before touching a config file. Everything below was executed on Python 3.13.5 with PyJWT 2.15.1 (and, where noted, 2.3.0), aarch64 Linux, OpenSSL 3.5.6. Transcripts are pasted from that run.

TL;DR — the message is a verdict, not a diagnosis

• The signature is computed over the ASCII bytes header.payload — both base64url segments, without padding, joined by a dot. Not over the claims, not over the JSON, not over the decoded header. Any change to either segment invalidates it, including a proxy that re-serialises the header JSON with spaces.
• A trailing newline in the secret is the most common cause I meet in production, and it is invisible: a 30-byte key instead of 29, one InvalidSignatureError, identical config files on both sides. Measured below, then turned into a startup failure instead of a silent strip.
• Base64 is the second. A 40-byte key where the signing side used 29 confirms in one line that one service decodes the secret and the other does not. Pick one convention and encode it in the variable name.
• Not every failure is this error. A truncated token raises DecodeError: Not enough segments; an algorithm mismatch raises InvalidAlgorithmError: The specified alg value is not allowed; an RSA public key offered as an HMAC secret raises InvalidKeyError. Reading the real class saves an hour.
• An empty third segment raises InvalidSignatureError, not DecodeError. header.payload. is a well-formed three-segment token with a zero-byte signature.
• PyJWT 2.15.1 refuses HMAC verification when the key is an asymmetric PEM — the algorithm-confusion attack dies at the library, not at your code. Keep the guard by always passing an explicit algorithms= list; do not disable it to "make it work".
• Key rotation is where this error is legitimate. A token signed with last month's key and verified with this month's key should fail — so select the key by kid from a two-entry map and keep both keys live for one token lifetime.

Cannot assign requested address (EADDRNOTAVAIL) at bind()

By Mara Voss · C2CZ · 16 October 2026 · Last verified: 2026-10-16

Mara Voss covers Docker, system limits and reverse proxies at C2CZ. Author page · All authors.

Your service refuses to start with Cannot assign requested address and you go looking for a port conflict — there isn't one. That message is EADDRNOTAVAIL (errno 99), printed when a process calls bind() on an IP address the kernel does not consider local; the port-conflict error is a different errno entirely (EADDRINUSE, 98). Every tutorial that answers this with "kill whatever is on the port" is answering the wrong question.

The setup is always the same: a service configured with a specific address instead of the wildcard — --bind 100.64.1.4 for a Tailscale address, --bind 172.18.0.7 for a container IP, listen 10.10.0.2:8443; for an interface that is up on the primary node but not the replica. On a healthy host it binds fine. Where that address is not (yet) assigned — tunnel not up, interface DOWN, container network not created when the process starts — the kernel refuses before it inspects the port. Below: the measured behaviour of each case, the rule the kernel applies, and the two fixes — bind the wildcard, or wait for the address.

TL;DR — errno 99 is not errno 98

• EADDRNOTAVAIL / errno 99, "Cannot assign requested address" — the address is not local to this network namespace. Measured here for 172.18.0.7, 10.0.0.1, 100.64.1.1 and 8.8.8.8. Nothing to kill; the port is irrelevant.
• EADDRINUSE / errno 98, "Address already in use" — the address is local, but the port is taken. This is the one that ss -ltnp and SO_REUSEADDR are for.
• Address locality is checked before the port. Measured: with 0.0.0.0:18080 occupied by another process, a bind to 172.18.0.7:18080 still returned errno 99, not 98. A port-conflict hypothesis sends you hunting in the wrong place.
• SO_REUSEADDR does not help. Set it to 1 and bind a non-local address: still errno 99. It only reopens TIME_WAIT ports (measured below).
• Loopback is a whole /8, not one address. localhost is whatever you configure, and binding 127.0.0.2 succeeds on Linux because the kernel marks 127.0.0.0/8 local — a trap when a container or test harness assumes "only 127.0.0.1 is usable".
• The fix is a decision, not a patch: (a) did you mean to bind a specific address at all? — usually no: bind 0.0.0.0 and let the firewall decide. (b) Do you genuinely need the tunnel address? — then retry the bind after the interface exists, and treat errno 99 as transient, errno 98 as terminal. The retry helper is further down and was executed to prove both paths.
• net.ipv4.ip_nonlocal_bind=1 is the documented escape hatch and the wrong default: it lets a process on one host silently complete for a VIP that is supposed to live on another. Value on the verification host: 0 (measured, not settable rootless here).

importKey on a PEM secret: the Workers ES256 fix

By Alex Rivera · C2CZ · 15 October 2026 · Last verified: 2026-10-15

Alex Rivera covers Cloudflare Workers and the serverless edge at C2CZ. Author page · All authors.

Store an IdP's public key as a Worker secret and call crypto.subtle.importKey("spki", PEM, ...), and the call throws before it touches your token. The message is a TypeError about the second argument that names neither PEM nor base64: you passed a string, and importKey accepts BufferSource or a JsonWebKey object. Nothing downstream — signature check, exp, iss — ever runs.

I hit this on a service verifying ES256 access tokens at the edge with a P-256 key pasted from the IdP's JWKS screen. Below is the twelve-line decoder that fixes it and the verifier I shipped: algorithm pinned before any crypto, key cached per isolate, six token shapes that must be rejected. Everything numeric was executed here — Node v26.5.1 webcrypto, the real Workers engine via workerd 2026-10-15 (npm [email protected], compatibilityDate = "2026-09-01"), OpenSSL 3.5.6 — and every transcript is pasted from those runs.

TL;DR — what fails, what fixes it

• A PEM is text; importKey wants bytes. The PEM string gives TypeError: Failed to execute 'importKey' on 'SubtleCrypto': parameter 2 is not of type 'BufferSource or JsonWebKey'. Passing just the base64 body changes nothing — still a string.
• The fix: strip the envelope, atob() the base64, copy into a Uint8Array, pass those bytes. atob is a global in Workers and Node 22+, so no node:buffer import and no compat flag. A P-256 SPKI public key is 91 DER bytes; the correct call returns public/ECDSA/P-256.
• Wording differs per runtime, the failure does not. The same call reads parameter 2 is not of type 'BufferSource or JsonWebKey' under workerd and 2nd argument is not instance of ArrayBuffer, Buffer, TypedArray, or DataView under Node v26.5.1. Do not grep logs for one fixed sentence.
• TypeError = wrong shape, DataError = wrong bytes. Truncated DER → DataError: Invalid SPKI input.; the right DER imported under the wrong algorithm object → DataError: Input was not an RSA key.
• A JsonWebKey object is the other valid path. importKey("jwk", jwkObject, ...) works; importKey("spki", jwkObject, ...) throws TypeError: Import data provided for buffer-based import formats must be a buffer source.; a JWK still in string form throws the PEM TypeError.
• Pin alg before verifying. Measured here: an HS256 token MACed with the public-key bytes is accepted by a verifier that trusts header.alg (true) and rejected by the one on this page (unsupported alg: HS256).

ENOSPC: System limit for number of file watchers reached

By Noah Lindqvist · C2CZ · 14 October 2026 · Last verified: 2026-10-14

Noah Lindqvist covers hardware, processes and Unix internals at C2CZ. Author page · All authors.

ENOSPC: System limit for number of file watchers reached has nothing to do with free disk space. The kernel returns ENOSPC from inotify_add_watch() when the per-user watch counter is full, and the dev server that reports it is usually the victim, not the cause. On this host the entire budget is two integers — fs.inotify.max_user_instances = 128 and fs.inotify.max_user_watches = 188727 — both charged per real user ID, both shared silently by every process you own, and both easy to measure with nothing but /proc.

Everything numeric below was executed on this machine: Linux 6.17 (arm64), uid 10000, Python 3.13.5, Node v26.5.1, Docker daemon absent. The lab calls inotify_init1(), inotify_add_watch() and read() through ctypes, so the error codes are the ones the kernel actually returned, and every block is pasted from that run rather than paraphrased.

TL;DR — two counters, one UID, measured

• Measured ceilings on this host: max_user_instances=128, max_user_watches=188727, max_queued_events=16384. All three are per real user ID, which is the detail that explains most of the confusion around this error.
• Watches are per inode, not per directory, and a directory watch is not recursive. Adding 1000 file watches took the descriptor's watch count from 1 to 1001 (measured in /proc/self/fdinfo), and touching a file one level below a watched directory produced exactly zero events.
• Instances run out long before watches do. A process opened 122 inotify instances and then failed on number 123 with errno=24 (Too many open files) — EMFILE. Six instances were already held by other processes of the same uid: 6 + 122 = 128.
• Node's fan-out is cheap in instances and expensive in watches. 200 fs.watch() calls by Node v26.5.1 occupied 1 instance and 200 watches, visible as a single fd with 200 inotify wd: lines.
• Audit before you raise. The budget is readable from /proc/<pid>/fdinfo/<fd> for every process you own; the auditor at the end of this page prints the per-process table and the percentage of each ceiling you are burning.
• Raising the ceiling is a sysctl, and it does not help a leak. fs.inotify.max_user_watches is a global kernel knob that every container on the host shares — raising it hides the leak and keeps the process that never closes a descriptor happily burning the new headroom.

error: gpg.ssh.allowedSignersFile needs to be configured

By Kai Tanaka · C2CZ · 13 October 2026 · Last verified: 2026-10-13

Kai Tanaka covers SSH, TLS and applied cryptography at C2CZ. Author page · All authors.

error: gpg.ssh.allowedSignersFile needs to be configured and exist for ssh signature verification means git cannot check your signatures, not that they are missing. The same run that prints that line on a signed commit will also print No signature a line later, which is the single most misleading message in git's signing stack: the commit is signed, the signing key was fine, and the only missing piece is the file that lists which keys you trust. Everything quoted on this page was executed on this host — OpenSSH 10.0p2, git 2.47.3, arm64 Linux 6.17 — against a throwaway repository and three throwaway keys, and every block below is pasted from that run, not paraphrased.

TL;DR — four states, four different messages

• No gpg.ssh.allowedSignersFile configured. git log --show-signature prints error: gpg.ssh.allowedSignersFile needs to be configured and exist for ssh signature verification and then No signature. git log --format=%G? reports U. The commit object still contains a complete BEGIN SSH SIGNATURE block — proof below.
• File configured, key or principal not listed. git prints Good "git" signature with ED25519 key SHA256:… followed by No principal matched. and exits 1 from git verify-commit. The signature is cryptographically valid and still untrusted — that is the distinction %G? encodes as U.
• File configured, key listed, all good. Good "git" signature for ops@build-01 with ED25519 key SHA256:…, %G? = G, git verify-commit exits 0.
• Verifying a commit by hand needs two removals, not one. The armour inside a commit carries git's gpgsig header prefix on its first line and a single leading space on every continuation line. Keep the prefix and the parser stops with missing header; drop the prefix but keep the fold and it stops with missing footer. Remove both and the same commit verifies clean.
• The operational trap. The options field in an allowed_signers entry is comma-separated. Write valid-after="…" valid-before="…" with a space and the field ends after the first option: the line is rejected as invalid key on OpenSSH 10.0p2, in both orders, and every commit in your pipeline stops verifying. valid-after="…",valid-before="…" is one field and enforces both bounds together.
• The fix is three config lines plus a file, and a CI verifier that checks signatures with ssh-keygen -Y verify directly — full script at the end, exit codes included, because ssh-keygen -Y verify writes its failure line to stdout, not stderr.

"socket hang up" (ECONNRESET): who closed the socket?

By Mara Voss · C2CZ · 12 October 2026 · Last verified: 2026-10-12

Mara Voss covers Docker, system limits and reverse proxies at C2CZ. Author page · All authors.

Error: socket hang up with code=ECONNRESET is Node telling you that the connection it was using died before a complete response arrived — and the string never says who killed it. Your process did not hang, nothing crashed, and the client did exactly what it promises: it wrote a request onto a socket, asked for a response, and the other end went away. Because the message names no peer, no address and no phase of the exchange, the same one line covers a dying container, a proxy that closed an idle upstream connection, a certificate-layer teardown and a load balancer rotating a backend. Everything quoted below was executed on this host — arm64 Linux 6.17, Node v26.5.1, Python 3.13.5, curl 8.14.1 — against the small origins built for this page, and every block printed byte-identical output on two consecutive runs.

TL;DR — three branches, three different fixes

• A reset before a single response byte. The peer's process went away: crash, docker restart, health-check kill, OOM. Node reports Error: socket hang up with code=ECONNRESET, curl exits 52 with Empty reply from server, Python raises http.client.RemoteDisconnected. Fix the lifetime of the peer, not the socket.
• The response had already started. Then it is not a hang-up at all but a truncated body: Node emits aborted on the response, curl exits 18, Python raises IncompleteRead. Different layer, different fix — buffering, a time limit, or a shutdown that does not drain.
• A graceful close on an idle kept-alive connection — the branch that never errors. A proxy, a tunnel or a restarted origin drops the idle connection; your client writes the next request onto a socket whose peer is already gone; nothing is thrown. Measured here: the first request returns status=200, the second is written into a dead socket and produces no error until the client's own timeout fires.
• The fix has two mandatory halves: a response timeout on every request, because a dead pooled socket is otherwise a black hole, and one retry on a fresh connection for methods that are safe to repeat. Both are measured below, and the client itself is the fourth block.

Disallowed operation called within global scope: the fix

By Alex Rivera · C2CZ · 11 October 2026 · Last verified: 2026-10-11

Alex Rivera covers Cloudflare Workers and the serverless edge at C2CZ. Author page · All authors.

Disallowed operation called within global scope is what the Workers runtime throws when module-scope code starts asynchronous I/O, sets a timer, or generates a random value — and it does not throw it from your handler: it throws it while loading your Worker, before any request exists. One crypto.randomUUID() or setTimeout() above your export default is enough to take the whole service down: the runtime logs Uncaught Error: Disallowed operation called within global scope…, never binds its socket, and every request dies before a handler runs. The same rule has two quieter failure modes that are worse than the crash — a Date.now() at module scope that returns 0 with no error at all, and an unawaited fetch() whose rejection nobody notices until the cache it filled is read by every subsequent request.

Every string quoted below was executed on this host — arm64 Linux 6.17, Node v26.5.1, the real Workers runtime workerd 2026-09-15 with compatibilityDate = "2026-09-01" — by the code shown, in the order shown. The harness starts one runtime process per case so a log line can never be attributed to the wrong module, and it was run twice; the two outputs were identical. Platform-level claims (the error-code family, what Cloudflare itself says about the top-level scope) are quoted from Cloudflare’s Errors and exceptions reference and labelled as such.

TL;DR — which branch are you in?

• Four shapes are fatal at load time, and all four are one line. await fetch(…), setTimeout(…), crypto.randomUUID() and crypto.getRandomValues(…) at module scope each produce the same Uncaught Error and the same dead service. getRandomValues is the surprise: it is synchronous, and it is still refused, so the runtime gates the API, not the promise.
• The failure is at load, not at request. The socket is never bound: curl reports 000 on every attempt. Cloudflare documents this class as a startup failure — the 10021 family, which covers “attempts to load and run the top-level scope… before your Worker’s handler is invoked”.
• Two quieter shapes load fine and serve 200, which is worse. A bare fetch(…) at module scope whose rejection is unhandled loads with an empty log. An unawaited async IIFE with a try/catch stores that error into a variable and reports it as clean data.
• The clock is the nastiest one. Date.now() at module scope returns 0; new Date() returns 1970-01-01T00:00:00.000Z. No error, no log line — a constant that every request will happily use for a token expiry, a cache key or a TTL.
• The rule is narrower than “no work at module scope”. await crypto.subtle.digest(…), await Promise.resolve(42) and Math.random() at module scope all work. Only asynchronous I/O, timers and crypto random values are refused.
• Bindings are not globals. Referencing GREETING or env at module scope is Uncaught ReferenceError: GREETING is not defined — also a startup failure, also the whole service. Bindings arrive as the handler’s second argument.
• A memoised promise can poison every later request. One rejection is replayed for the lifetime of the isolate: measured 502 on request 1 and request 2. Clear it in a .catch().

Argument list too long (E2BIG): fix execve argv limits

By Noah Lindqvist · C2CZ · 10 October 2026 · Last verified: 2026-10-10

Noah Lindqvist covers hardware, processes and Unix internals at C2CZ. Author page · All authors.

Argument list too long is not a filesystem error and not a shell option: the kernel's execve() refused the argument list before your program ever started. Nothing in the command ran, no partial work was done, and the exit status the shell reports is 126 — found but not executable by exec.

Two integers decide whether a command starts. Every argument and every environment string is copied into the new process's stack, and the kernel counts them together: strings, their NUL terminators, and eight bytes of pointer per string must fit under ARG_MAX. On this host that is 2,097,152 bytes, but a command that carries a fat environment cannot spend most of it, and no single argument may exceed 32 pages. Exceed either number and execve() returns E2BIG (errno 7).

This article is the measurement first and the fixes second. Everything below was executed on arm64 Linux 6.17.0-1019-oracle with bash 5.2.37, CPython 3.13.5, Node v26.5.1, GNU findutils and coreutils; every transcript is the unedited output of the block above it, captured twice and diffed so the numbers repeat on your machine.

TL;DR — the numbers, and the three fixes

• E2BIG (7) comes from execve(), not from the program. There is no partial run and no output — the child never existed. bash reports exit 126; Python raises [Errno 7] Argument list too long; Node returns error.code === "E2BIG".
• A single argument is capped at 32 pages. Measured: 131,071 bytes is accepted, 131,072 is refused, whatever the rest of the argv looks like (MAX_ARG_STRLEN = 32 × 4096).
• argv and the environment share one budget. Measured: 2,088,960 bytes of arguments fit with an empty environment; the same binary accepted only 122,880 bytes once the environment grew to 1.9 MiB. A long PATH, a CI matrix, or mounted secrets all shrink your argument budget before your command runs.
• The shell builtin hides the bug. echo * over 40,000 files prints nothing and exits 0; /bin/echo * on the same glob dies with exit 126. Only the second one calls execve().
• Three fixes, in order of preference: xargs -0 -n for existing pipelines, find ... -exec cmd {} + for file walks, and a runner that sizes each batch from the live budget when the environment is not yours (cron, CI, docker exec).
• Log the errno, not "no output". A failed execve() and a command that legitimately prints nothing look identical in a cron mailer.