debugging network issues

Other Community

Skill notes

Triage first — is this a known domain?

Before applying the general methodology below, check whether the symptom points at a stack that already has a dedicated skill in this repo. Those carry the domain-specific symptom→cause→fix tables this skill deliberately stays general about — start there, and come back here for methodology if the root cause turns out to be elsewhere. If the symptom is… · Start with macOS Tailscale ⨯ proxy/VPN conflict (Shadowrocket / Clash / Surge): tailscale ping works but SSH/curl/git fails, Connection closed by 198.18.x.x, TUN DNS hijack, ~60s getaddrinfo resolver stall · tunnel-doctor

1. Evidence over assumption

If you cannot point to a concrete artifact — log line, pcap frame, probe output, metric sample — you are guessing, not diagnosing. Before stating "X is the cause", require yourself to name the direct evidence. If it does not exist yet, add instrumentation (see references/instrumentation-patterns.md) or capture it (see references/packet-capture-recipes.md) before continuing.

2. Falsification over confirmation

N independent sources "confirming" a hypothesis does not make it true. One falsifying observation rules it out. Before acting on a hypothesis, answer: "What observation would make me abandon this hypothesis?" If the answer is "nothing" or "I cannot think of one", the hypothesis is unfalsifiable and must not drive the investigation. If the answer is concrete, go look for that observation before committing to action.

3. Layered isolation

Multi-hop systems (client → CDN → LB → reverse proxy → app → upstream) concentrate bugs at the seams between layers. When a symptom could plausibly come from several layers, do not reason about which layer; test. The canonical technique: run the same logical request through three or more paths that differ by exactly one hop, then compare where the symptom appears. This resolves in minutes what stacking hypotheses cannot resolve in hours. See references/layered-isolation-experiment.md.

4. Counter-review before committing

Before committing to a root cause or shipping a fix, have independent reviewers challenge the conclusion — not confirm it. Agents are good at surfacing risks a single investigator did not think of; they are bad at weighing them. Apply the four-question filter (see references/counter-review-pattern.md) to every finding before it shapes action.

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Other
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@daymade · no license declared
Token usage
Heavy
Setup complexity
Manual integration
External API key
Not required
Operating systems
macOS · Linux · Windows · Docker
Runtime requirements
Node.js · Docker
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
  • Env read
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

01
Step 0: Scope

A tight scope is the difference between a 20-minute investigation and a 5-hour one. Before looking at anything, extract: Exact error string (copy-paste, not paraphrase). `socket closed` is not the same as `ECONNRESET` is not the same as `HTTP/2 RST_STREAM INTERNAL_ERROR (err 2)`. Exact timestamps (ISO-8601 with timezone, not "yesterday evening") Reproducibility (every time / intermittent / only specific users)

02
Step 0.5: Verify the premise

Before investing in a full investigation, confirm the reported symptom is actually happening — not just inferred from downstream effects or user frustration. One cheap direct observation beats hours spent investigating a non-problem. Ask: "What direct evidence shows this symptom is real?" If the user reports "timeout at 130s": is that from a timestamped log, a browser network panel, or a recollection?

03
Step 0.6: Upload-timeout vs processing-timeout for large POST bodies

For CDN-fronted `POST`/`PUT` endpoints with large bodies, the most common misdiagnosis is blaming backend slowness when the real problem is time-to-upload-body exceeding the CDN/proxy origin timeout. Apply this sub-checklist when the symptom is a 524/522/504 on a request with `Content-Length > ~500 KB`: Locate the edge/reverse-proxy access log (Caddy, nginx, Envoy, Cloudflare Logpush). Compare `bytes_read` (or equivalent) to `Content-Length`:

04
Step 0.7: Throughput collapse — when nothing errors and everything is slow

The recipe above (0.6) handles a *timeout* — an error code you can grep for. This one handles the family with no error at all: every request succeeds, every health check is green, and the system is unusable because bytes arrive at a fraction of the expected rate. It is the failure mode most likely to be misdiagnosed for hours, because the entire error-driven toolkit has nothing to bite on. Why the usual signals go green — each of these was observed simultaneously with a 100×-degraded link:

05
Step 1: Gather direct evidence at every hop

Before framing hypotheses, collect: Server-side logs at every hop in the request path Client-side logs (browser devtools HAR, CLI debug log, SDK traces) Metrics over the incident window (RPS, latency, error rate, connection count, CPU/mem) Distributed trace if available

06
Step 2: Hypotheses with falsifiers and threat-model boundaries

List three or more plausible causes. For each, write three sentences: What would confirm it?

07
Step 3: Decisive experiment

For network-layer problems, the default is layered isolation: three paths differing by exactly one hop. Example for a CDN-fronted service: Path · Route · Rules out if it passes A · Full path via CDN · Nothing — this is the failing baseline B · `--resolve` to origin IP (bypass CDN) · CDN layer C · Server loopback (bypass CDN + LB) · CDN + LB

08
Step 4: Instrumentation when needed

If the decisive experiment requires an observation that cannot currently be made, add it — do not skip it. The canonical pattern is env-gated instrumentation that: Defaults off (zero runtime cost in steady state) Turns on via one environment variable, without code changes Writes greppable log tags (`[SSE-CHUNK] ts=... req=... bytes=...`) Ships into production permanently — future incidents reuse it

09
Step 5: Execute and record

Run the experiment once, fully documented: command, environment, inputs, observed outputs, wall-clock timestamps. Compare against the prediction made in Step 2. If actual matches predicted, the hypothesis is calibrated. If not, the hypothesis is wrong — do not rescue it with ad-hoc auxiliary hypotheses ("oh, but maybe X also interferes..."). Return to Step 2 and write new hypotheses from scratch.

10
Step 6: Counter-review

Before committing to a root cause or shipping a fix, spawn independent reviewers to challenge the conclusion. Give them the same evidence, ask them to falsify, not confirm. Apply the four-question filter to each finding they raise: Probability — will this actually happen? Cost — what is the cost of fixing versus ignoring? Realistic scenario — does this apply to the user's actual business case? Verification — can I cheaply confirm or refute this?

11
Step 7: Fix and verify

Apply the fix. Rerun the same decisive experiment from Step 3. Confirm the symptom no longer reproduces with the same setup that was reliably producing it. If the pre-fix state can no longer be reproduced after the fix, the fix cannot be proven — figure out why the repro was lost before declaring victory.

12
Step 8: Document wrong turns

The wrong turns in the investigation are more valuable than the right answer. Write an incident report capturing: Symptom + direct evidence Each hypothesis tried + how it was falsified Decisive experiment design + result Fix + verification

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.