The verification ladder

The AI bug catalog named a tool for each of its 24 patterns — mostly “none — only a human.” This lesson runs every tool that is real, for real, against real Node.js code, and quotes exactly what came back: no invented output, no imagined rule names. None of these tools ship with this repo — each was installed in a disposable scratch project (npm install --save-dev for the npm packages, brew install osv-scanner for the one that isn’t one) and run there, exactly as a reviewer would set one up for a project that does not have it yet.

Rung 1: node --test with coverage

The built-in test runner needs no install at all. Given math.cjs exporting add and subtract, and a test file that only exercises add:

// @skip-verify illustrative -- ./math.cjs is a separate file, shown only to name the setup
const test = require('node:test');
const assert = require('node:assert/strict');
const { add } = require('./math.cjs');

test('adds two numbers', function () {
  assert.equal(add(2, 3), 5);
});

Node.jsOpen in StackBlitz ▸

import assert from 'node:assert/strict'; function add(a, b) { return a + b; } assert.equal(add(2, 3), 5); console.log('assertion passed');

Output (verified on Node 24)

assertion passed

Run for real with node --test --experimental-test-coverage:

✔ adds two numbers (0.5ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 58.8
ℹ start of coverage report
ℹ ----------------------------------------------------------
ℹ file      | line % | branch % | funcs % | uncovered lines
ℹ ----------------------------------------------------------
ℹ math.cjs  |  66.67 |   100.00 |   50.00 | 5-7
ℹ ----------------------------------------------------------
ℹ all files |  66.67 |   100.00 |   50.00 |
ℹ ----------------------------------------------------------
ℹ end of coverage report

(Per-test timing varies run to run; the coverage numbers do not.) subtract is never called from the test file, and the coverage report says so directly: 66.67% line coverage, subtract’s lines listed as uncovered. --experimental-test-coverage is still an experimental flag on Node 24.13 — stable enough to rely on, unstable enough that its exact report format could still change in a later release.

Rung 2: --trace-warnings, --unhandled-rejections=strict, --trace-uncaught

--trace-warnings turns a bare warning into a full stack trace pointing at exactly where the problem was created. Run against the listener-leak pattern from the AI bug catalog:

listener count: 11
(node:45399) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 data listeners added to [EventEmitter]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
    at genericNodeError (node:internal/errors:985:15)
    at wrappedFn (node:internal/errors:539:14)
    at _addListener (node:events:582:17)
    at EventEmitter.addListener (node:events:600:10)
    at handleRequest (file:///.../listener-leak.mjs:10:11)
    at file:///.../listener-leak.mjs:13:30
warning name: MaxListenersExceededWarning

Without the flag, the warning still prints — just without the stack, leaving “where did this listener actually get added?” as a manual search through the codebase instead of a line number.

--unhandled-rejections=strict has been the default behavior since Node 15 — running it explicitly against the floating-promise pattern produces the same crash either way:

save queued
Error: missing id
    at saveRecord (...)
Node.js v24.13.0

The flag matters for code that still runs on an ancient Node version, or for a codebase that has deliberately overridden the default back to warn; on a current Node 24.13 install, the crash happens regardless.

--trace-uncaught adds one more section, Thrown at:, on top of Node’s default uncaught-exception output — pointing at exactly the line the error was thrown from, separate from where it was ultimately caught (or not caught) further up the stack. Against the error-event-with-no-listener pattern, the default output already shows Emitted 'error' event at:; --trace-uncaught adds a second Thrown at: block with the same location for this simple case, but the two can differ in code where an error is thrown in one place and only surfaces to the top level somewhere else entirely.

Rung 3: eslint-plugin-n + eslint-plugin-promise, run for real

Installed in the scratch project: [email protected], [email protected], [email protected]. The flat config, using each plugin’s own exported flat preset:

// @skip-verify eslint config, not a runnable program
import n from 'eslint-plugin-n';
import promise from 'eslint-plugin-promise';

export default [
  n.configs['flat/recommended-module'],
  promise.configs['flat/recommended'],
  {
    rules: {
      'no-eval': 'error',
    },
  },
];

Run against a target file built to combine several AI bug catalog patterns at once:

// @expect-throw
import fs from 'node:fs';

const data = fs.readFileSync('/etc/hosts');

function fetchUser(id) {
  return new Promise(function (ok, bad) {
    ok({ id: id });
  });
}

function handleUser(id) {
  fetchUser(id).then(function (user) {
    console.log(user);
  });
}

function chained(id) {
  return fetchUser(id).then(function (user) {
    fetchUser(user.id + 1).then(function (next) {
      console.log(next);
    });
  });
}

handleUser(1);
chained(2);

eval('console.log(1)');

process.exit(1);

new Buffer(10);

npx eslint bad-file.mjs, real output:

  6:32  error    Promise constructor parameters must be named to match "^_?resolve$"                          promise/param-names
  6:36  error    Promise constructor parameters must be named to match "^_?reject$"                           promise/param-names
  12:3  error    Expected catch() or return                                                                   promise/catch-or-return
  12:22 error    Each then() should return a value or throw                                                   promise/always-return
  18:29 error    Each then() should return a value or throw                                                   promise/always-return
  19:5  error    Expected catch() or return                                                                   promise/catch-or-return
  19:28 warning  Avoid nesting promises                                                                       promise/no-nesting
  19:33 error    Each then() should return a value or throw                                                   promise/always-return
  28:1  error    eval can be harmful                                                                          no-eval
  30:1  error    Don't use process.exit(); throw an error instead                                             n/no-process-exit
  32:1  error    'new Buffer()' was deprecated since v6.0.0. Use 'Buffer.alloc()' or 'Buffer.from()' instead   n/no-deprecated-api

✖ 11 problems (10 errors, 1 warning)

Seven distinct rules fired on one file: promise/param-names, promise/catch-or-return, promise/always-return, promise/no-nesting, no-eval, n/no-process-exit, n/no-deprecated-api. Notice what did not fire: fs.readFileSync on line 3 (pattern 6, synchronous work in a hot path) has no rule against it at all — reading it as risky requires knowing it runs inside a request handler, which is exactly the kind of context a linter, which sees one file at a time, does not have.

Rung 4: npm audit and osv-scanner

Against a scratch project pinned to [email protected] (npm audit, real output):

# npm audit report

lodash  <=4.17.23
Severity: critical
Prototype Pollution in lodash - https://github.com/advisories/GHSA-fvqr-27wr-82fm
Command Injection in lodash - https://github.com/advisories/GHSA-35jh-r3h4-6jhm
Prototype Pollution in lodash - https://github.com/advisories/GHSA-4xc9-xhrj-v574
Prototype Pollution in lodash - https://github.com/advisories/GHSA-jf85-cpcp-j695
Prototype Pollution in lodash - https://github.com/advisories/GHSA-p6mc-m468-83gw
Regular Expression Denial of Service (ReDoS) in lodash - https://github.com/advisories/GHSA-29mw-wpgm-hmr9
Regular Expression Denial of Service (ReDoS) in lodash - https://github.com/advisories/GHSA-x5rq-j2xg-h7qm
lodash vulnerable to Code Injection via `_.template` imports key names - https://github.com/advisories/GHSA-r5fr-rjxr-66jc
lodash vulnerable to Prototype Pollution via array path bypass in `_.unset` and `_.omit` - https://github.com/advisories/GHSA-f23m-r3pf-42rh
Lodash has Prototype Pollution Vulnerability in `_.unset` and `_.omit` functions - https://github.com/advisories/GHSA-xxjr-mmjv-4gpg
fix available via `npm audit fix --force`
Will install [email protected], which is outside the stated dependency range
node_modules/lodash

1 critical severity vulnerability

The same package-lock.json, osv-scanner scan source -L package-lock.json (osv-scanner 2.6.0, installed via brew install osv-scanner):

Total 1 package affected by 8 known vulnerabilities (1 Critical, 2 High, 4 Medium, 0 Low, 1 Unknown) from 1 ecosystem.
8 vulnerabilities can be fixed.

+-------------------------------------+------+-----------+---------+---------+---------------+
| OSV URL                             | CVSS | ECOSYSTEM | PACKAGE | VERSION | FIXED VERSION |
+-------------------------------------+------+-----------+---------+---------+---------------+
| https://osv.dev/GHSA-29mw-wpgm-hmr9 | 5.3  | npm       | lodash  | 4.17.4  | 4.17.21       |
| https://osv.dev/GHSA-35jh-r3h4-6jhm | 8.1  | npm       | lodash  | 4.17.4  | 4.17.21       |
| https://osv.dev/GHSA-4xc9-xhrj-v574 |      | npm       | lodash  | 4.17.4  | 4.17.11       |
| https://osv.dev/GHSA-f23m-r3pf-42rh | 6.9  | npm       | lodash  | 4.17.4  | 4.18.0        |
| https://osv.dev/GHSA-fvqr-27wr-82fm | 6.5  | npm       | lodash  | 4.17.4  | 4.17.5        |
| https://osv.dev/GHSA-jf85-cpcp-j695 | 9.1  | npm       | lodash  | 4.17.4  | 4.17.12       |
| https://osv.dev/GHSA-p6mc-m468-83gw | 7.4  | npm       | lodash  | 4.17.4  | 4.17.19       |
| https://osv.dev/GHSA-x5rq-j2xg-h7qm | 6.5  | npm       | lodash  | 4.17.4  | 4.17.11       |
+-------------------------------------+------+-----------+---------+---------+---------------+

Both tools found the same underlying advisories from the same lockfile; osv-scanner additionally reports CVSS scores and the specific fixed version per advisory, and reads the OSV database directly rather than npm’s own advisory feed. Neither tool has anything to do with this catalog’s pattern 24 (exec/eval injection in your own code) — both only know about vulnerabilities in packages you depend on, not bugs you wrote yourself.

Rung 5: --cpu-prof and autocannon against the Production Server

Against the exact server from the Production Server lesson, listening on a real port, autocannon -c 10 -d 4 on /health:

┌─────────┬──────┬──────┬───────┬──────┬─────────┬─────────┬──────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%  │ Avg     │ Stdev   │ Max  │
├─────────┼──────┼──────┼───────┼──────┼─────────┼─────────┼──────┤
│ Latency │ 0 ms │ 0 ms │ 0 ms  │ 0 ms │ 0.01 ms │ 0.05 ms │ 6 ms │
└─────────┴──────┴──────┴───────┴──────┴─────────┴─────────┴──────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬───────────┬──────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg       │ Stdev    │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼───────────┼──────────┼─────────┤
│ Req/Sec   │ 60,479  │ 60,479  │ 67,007  │ 71,487  │ 67,572    │ 4,487.68 │ 60,456  │
└───────────┴─────────┴─────────┴─────────┴─────────┴───────────┴──────────┴─────────┘

270k requests in 4.01s, 58.7 MB read

Bucketed for reproducibility (the exact numbers move run to run, the bucket does not): sub-millisecond latency, roughly 60k–75k requests/second on this machine for a handler that does no real work beyond JSON.stringify. Running the same load with node --cpu-prof --cpu-prof-dir=. --cpu-prof-name=prod.cpuprofile server.cjs produces a real .cpuprofile file (loadable in Chrome DevTools’ Performance panel, or inspected as JSON directly); summing hit counts by function name from this run’s actual profile:

(idle)               1899
writev                 973
(program)              229
writevGeneric           59
finish                  53
_storeHeader            45
nextTick                44
(garbage collector)     41

(idle) dominates — expected for a health-check handler with no CPU-bound work — and the next-hottest real frames are all inside Node’s own HTTP writing path (writev, writevGeneric, _storeHeader, finish), not application code. This is exactly the signature that would look different for pattern 6 (synchronous CPU work in a handler): a busy-loop or a large JSON.parse would show up as a hot, named application frame instead of disappearing into (idle) and the socket-write internals.

Rung 6: knip (unused exports)

Against a two-file scratch package where math.js exports add, subtract, and multiply, but only add is ever imported:

Unused exports (2)
subtract  function  src/math.js:5:17
multiply  function  src/math.js:9:17

knip finds this by tracing the actual import graph, not by pattern-matching text — it is the one tool on this ladder that answers a question none of the 24 catalog patterns are really about (“is this code reachable at all?”) rather than “is this reachable code correct?”

The pattern-to-rung table

#	Pattern	Rung that catches it
1	Forgotten await	None — only a human
2	forEach(async ...)	None — only a human
3	Floating promise / unhandled rejection	eslint-plugin-promise’s promise/catch-or-return; crash is Node’s own default
4	Promise.all unbounded concurrency	None — only a human (surfaced by autocannon/production load)
5	setTimeout(fn, 0) vs setImmediate/nextTick	None — only a human
6	Sync CPU work in a handler	None — only a human (surfaced by --cpu-prof/monitorEventLoopDelay)
7	res.end() called twice	None — only a human
8	Headers set after response sent	None — only a human
9	No request body limit	None — only a human (surfaced by autocannon)
10	JSON.parse with no try/catch	None — only a human
11	fetch without .ok/timeout check	None — only a human
12	Listener leak	--trace-warnings
13	error event with no listener	None — only a human
14	.pipe() with no error handling	None — only a human
15	String decoding across a chunk boundary	None — only a human
16	Buffer.concat inside a loop	None — only a human (surfaced by --cpu-prof)
17	process.env.PORT compared as a string	None — only a human
18	process.exit() mid-flush	None — only a human
19	require/import mixing, __dirname in ESM	None — only a human (the runtime error is immediate)
20	^0.x, no lockfile, npm install in Docker	npm audit/npm ci catch related drift, not the Dockerfile choice itself
21	Wrong exports field order	None — only a human
22	No assertion / stale mock / real-timer test	node --test, once a real assertion and the right mock target exist
23	console.log instead of structured logs	None — only a human
24	exec/eval injection	None — only a human (npm audit/osv-scanner catch vulnerable dependencies, not this)

One command for the whole ladder

node --test --experimental-test-coverage && \
  NODE_OPTIONS="--trace-warnings --unhandled-rejections=strict" node --test && \
  npx eslint . && \
  npm audit --audit-level=high

Four rungs, one shell line, &&-chained so a failure at any stage stops the rest — --cpu-prof/autocannon and knip are left out of the one-liner deliberately, since they need a running server and a full import graph respectively, not just a source tree to lint and test.

Mental Model

Every rung on this ladder answers a narrow, specific question — does the code do what the tests say, did anything warn, is the style consistent, are the dependencies known-vulnerable, is it fast enough, is anything unreachable — and none of them answer “is this code correct.” The pattern-to-rung table’s own numbers make that concrete: 21 of 24 catalog patterns have no automated rung at all.

Trade-offs

Choice	Benefit	Cost
Running the full ladder on every PR	Catches real, if narrow, classes of bugs automatically, every time	Six separate tools, several requiring network/CI setup (osv-scanner, autocannon needing a live server)
Running only node --test + eslint	Fast enough for a pre-commit hook	Misses dependency vulnerabilities, unreachable code, and performance regressions entirely
Treating a clean ladder run as “reviewed”	None — this is never a good trade	Every tool here is narrow by design; a clean run only proves the narrow things it checks were fine

Common Misconceptions

• “--experimental-test-coverage being experimental means it is unreliable.” The flag name signals the report format may still change, not that the coverage numbers it prints today are wrong — they came from real V8 coverage instrumentation, the same primitive Istanbul-based tools build on.
• “npm audit and osv-scanner are redundant since they found the same advisories here.” They read different databases (npm’s own advisory feed vs. the OSV aggregator) and can diverge on non-npm ecosystems or very recent advisories — running one is not a substitute for the other for every project.
• “A load test with autocannon proves the server has no bugs.” It proves this specific handler holds up under this concurrency and this payload shape on this machine — it says nothing about a different route, a larger body, or a genuinely CPU-bound handler.

Reading AI-generated code

Before trusting an AI assistant’s claim that “the tests pass” or “lint is clean,” reproduce it: the AI bug catalog’s pattern 22 exists precisely because a green node --test run can mean “no reachable assertion failed,” not “the code is correct.” Run the actual ladder above against the actual diff, read the actual rule names in the output, and check each hit against the pattern-to-rung table before deciding a tool has “handled” a concern the table says it does not cover.

🐛 Spot the bug

// PR description: "Added tests, lint is clean, ran npm audit -- all green."
function calculateDiscount(price, percent) {
return price - percent; // BUG: should be price - (price * percent / 100)
}

import { test } from 'node:test';
test('applies a discount', function () {
calculateDiscount(100, 10); // BUG: result never checked with an assertion
});

Show the answer

“Lint is clean” and “npm audit is green” are both true and both irrelevant here: no eslint rule in this course’s ladder checks arithmetic, and npm audit only scans dependencies, not this file’s own logic. The only rung that could have caught the actual bug — node --test — reports a pass too, because the test never asserts anything about the return value. All three tools genuinely ran, and all three genuinely say nothing about whether calculateDiscount is correct; only reading the function body against its name catches that 100 - 10 is being returned instead of 100 - (100 * 10 / 100).

🟢 The one-line takeaway

Six real tools, six narrow answers — node --test, --trace-warnings, eslint-plugin-n/promise, npm audit/osv-scanner, --cpu-prof/autocannon, and knip together still catch only 3 of this course’s 24 cataloged patterns automatically; the pattern-to-rung table is the map of exactly which ones, so nobody has to guess.

The eslint run in this lesson fires 7 distinct rules on an 18-line file, but says nothing about fs.readFileSync('/etc/hosts') on line 3. Why not?

eslint-plugin-n does not exist for synchronous fs calls Whether a synchronous call is dangerous depends on whether it runs inside a hot path serving concurrent requests -- context a linter examining one file in isolation does not have The file has a syntax error that stops eslint before reaching line 3 readFileSync is deprecated, but the config used here disables that rule

Why do npm audit and osv-scanner, run against the same lockfile pinned to [email protected], report the same underlying vulnerabilities but with some different details?

One of the two tools is reporting false positives They query different vulnerability databases (npm's own advisory feed vs. the OSV aggregator), so they can differ in extra details like CVSS scores even when they agree on the underlying advisories osv-scanner only works for Python projects npm audit and osv-scanner are actually the same tool with two different names

According to the pattern-to-rung table, which of these AI bug catalog patterns is the ONLY one caught by an eslint rule directly, rather than being "none -- only a human"?

Pattern 7 (double res.end()) Pattern 3 (a floating promise that later rejects), via eslint-plugin-promise's catch-or-return rule Pattern 15 (string decoding across a chunk boundary) Pattern 24 (exec/eval injection)

The autocannon run against the Production Server's /health route reports roughly 60k-75k requests/second on this machine. What does this number NOT tell you?

Whether the /health route itself can handle real concurrent traffic Whether a different route in the same server -- one that does real database queries or synchronous CPU work -- would perform anywhere close to this The approximate order of magnitude of throughput for this specific trivial handler That the server successfully handled the requests autocannon sent

A reviewer sees a PR with a node --test run that reports "1 passed, 0 failed" for a new function, and treats this as proof the function is correct. Based on this lesson and the AI bug catalog, what is the gap in that reasoning?

node --test never actually runs the code, only checks syntax A passing test only proves that whatever assertions the test actually contains did not fail -- if the test contains no assertion (or asserts the wrong thing), a pass proves nothing about correctness node --test results cannot be trusted on any machine other than the one that ran them There is no gap -- a passing test always proves correctness

Submit

Mark as complete