ntfy alert triage

Ordered triage for the anton alert pipeline: PrometheusRule → Alertmanager → AlertmanagerConfig route → webhook → self-hosted ntfy at ntfy.<tailnet>.ts.net → (optionally) ntfy.sh upstream relay → device. The 2026-05-05 cascade where a 40014 from ntfy turned into a 5-minute AlertmanagerClusterFailedToSendAlerts loop is the canonical failure mode this skill is built around.

Pipeline map (memorise this)

PrometheusRule                                Helm-managed via kube-prometheus-stack;
   │                                          Anton-authored rules live in
   │                                          kubernetes/apps/observability/kube-prometheus-stack/app/
   ▼
Alertmanager (alertmanager-kube-prometheus-stack-alertmanager-0)
   │  route: severity=critical → observability/ntfy/ntfy receiver
   │  everything else → "null" receiver
   │  (see AlertmanagerConfig at kubernetes/apps/observability/ntfy/app/alertmanagerconfig.yaml)
   ▼
webhook POST to URL from Secret ntfy-topic in observability ns
   │                                          (templated by ESO from 1Password ntfy/topic)
   ▼
self-hosted ntfy (Deployment ntfy in observability)
   │  base-url: https://ntfy.<tailnet>.ts.net
   │  upstream-base-url: https://ntfy.sh   ← iOS push relay (ADR 0026)
   │  attachment-cache-dir set since 2026-05-05 (configmap.yaml)
   ▼
device (browser at https://ntfy.<tailnet>/<topic>, or iOS via ntfy.sh poll trigger)

Decision tree — which question are we answering?

Step 1 — What just fired?

kubectl exec -n observability alertmanager-kube-prometheus-stack-alertmanager-0 \
  -c alertmanager -- wget -qO- \
  'http://localhost:9093/api/v2/alerts?active=true&silenced=false&inhibited=false' \
  | python3 -c "import json,sys; [print(a['labels'].get('alertname'),'|',a['labels'].get('severity'),'|',a['labels'].get('instance',''),'|','start:',a['startsAt'],'|','rcv:',[r['name'] for r in a['receivers']]) for a in json.load(sys.stdin)]"

Sort the active alerts by startsAt and find the one that matches the time the user was paged. Only alerts with rcv: containing observability/ntfy/ntfy actually went to ntfy; everything else routes to null.

Ground-truth check before reacting: a fresh-looking alert can be a metric/series artifact, not a real event. For NodeUnexpectedReboot specifically, check /proc/uptime directly via talosctl; cross-reference with the cluster-triage agent memory at .Codex/agent-memory/cluster-triage/reference_reboot_alert_disambiguation.md.

Step 2 — Why didn't it deliver?

Two failure classes — distinguish by alertmanager behaviour.

Class A: never reached Alertmanager — Prometheus didn't fire it, or the rule expression doesn't evaluate to truth, or the rule isn't loaded.

# Did Prometheus load the rule?
kubectl exec -n observability prometheus-kube-prometheus-stack-prometheus-0 \
  -c prometheus -- wget -qO- http://localhost:9090/api/v1/rules \
  | python3 -c "import json,sys; [print(g['name'],'/',r['name']) for f in json.load(sys.stdin)['data']['groups'] for r in f['rules'] for g in [f]]" \
  | grep -i <alertname>

# Is the expression returning samples right now?
# (Pull the expr from the PrometheusRule yaml and test it via /api/v1/query.)

Class B: reached Alertmanager but didn't deliver — receiver matchers excluded it, or webhook delivery failed.

# Receiver matchers — only severity=critical reaches ntfy in anton.
kubectl exec -n observability alertmanager-kube-prometheus-stack-alertmanager-0 \
  -c alertmanager -- wget -qO- http://localhost:9093/api/v2/status \
  | python3 -c "import json,sys,re; print(re.search(r'route:.*?inhibit_rules', json.load(sys.stdin)['config']['original'], re.S).group(0))"

If your alert has severity warning/info, it routes to null by design. To reach ntfy, either bump severity in the rule or broaden the AlertmanagerConfig matcher (see ADR 0026 — "easier to broaden than to silence"; reconsider widening to "everything except info").

Step 3 — Decode the delivery failure

This is the 2026-05-05 cascade flow. When AlertmanagerClusterFailedToSendAlerts is firing, the actual error is in the alertmanager pod logs, not the alert payload.

kubectl logs -n observability alertmanager-kube-prometheus-stack-alertmanager-0 \
  -c alertmanager --tail=200 | grep -iE 'error|fail|notify' | tail -20

Match the error to the table:

Inhibit while fixing: the cluster-failed-to-send alert will keep paging while you work. Silence it via amtool silence add alertname=AlertmanagerClusterFailedToSendAlerts --duration=30m -c <annotation>, or accept ~5-15 min of stale alerts after the fix lands.

Step 4 — Probe the pipeline with the ntfy CLI

ntfy is preinstalled on the operator workstation. Reference: https://docs.ntfy.sh/subscribe/cli/.

The webhook URL — including the secret topic — lives in the ntfy-topic Secret in observability. Never echo the URL to stdout or write it to a file (AGENTS.md hard rule). Pipe it directly:

# Read the URL into a shell variable WITHOUT printing it.
# (Single command; topic stays out of shell history if HISTCONTROL=ignorespace and you prefix with a space.)
 NTFY_URL=$(kubectl get secret -n observability ntfy-topic -o jsonpath='{.data.url}' | base64 -d)

Then split server + topic if you need them separately:

 NTFY_SERVER="${NTFY_URL%/*}"
 NTFY_TOPIC="${NTFY_URL##*/}"

Read recent deliveries (no long-running connection)

ntfy subscribe --poll --since 30m "${NTFY_URL}"

Use --since 1h / --since 12h for wider windows (ntfy cache-duration: 12h in anton's ConfigMap; older messages are gone). The --poll flag fetches and exits — never start a backgrounded ntfy subscribe from this skill.

Send a test publish

ntfy publish --title "anton triage probe" --priority default --tags test \
  "${NTFY_URL}" "$(date -u +%FT%TZ) probe from ntfy-alert-triage skill"

A successful publish prints the message ID and returns 0. A 40014 here means attachments are still disabled (see Step 3). A 401 means ACLs were added without updating the secret. A 404 on the topic means the topic name doesn't match the URL path.

Reproduce the 40014 (size-spill check)

The original cascade was triggered by Alertmanager grouping 2+ alerts into one webhook body that exceeded ~5 KiB. To verify attachments-cache works end-to-end:

ntfy publish "${NTFY_URL}" "$(python3 -c 'print("x"*8000)')"

Pre-fix this returned 40014; post-fix it succeeds and the message is stored as an attachment (visible in the ntfy web UI, not inline on iOS).

Verify the iOS upstream relay

iOS push goes via upstream-base-url: https://ntfy.sh (ADR 0026 — only message IDs transit; bodies stay on cluster). To check the upstream poll registration:

kubectl logs -n observability deploy/ntfy --tail=50 | grep -i upstream

A healthy line looks like Successfully forwarded message to upstream. Errors mean iOS won't get push notifications even when desktop browser delivery works.

Anton-specific reference

Hard rules (carry-overs from AGENTS.md)

• Never echo, log, or write the topic URL or secret value. Always pipe kubectl get secret ... -o jsonpath directly into the consuming command.
• Never edit *.sops.* files in plaintext; use sops <file> for round-trip.
• Never restart the alertmanager StatefulSet to "fix" a delivery problem — the failure is upstream of AM in 99% of cases. Restart ntfy if the ntfy pod is the suspect; restart AM only if its own logs say so.
• The Vector kernel sink (talos-log-sink) and its 30Gi PVC are unrelated to ntfy alerting; if both are firing alerts, treat them independently.

Hand-offs

• Rule expression is wrong / missing → observability-integrate
• ntfy pod itself is broken (CrashLoop, OOM, image pull) → anton-cluster-health layer 5, then debug-flux-reconciliation if it's a Flux apply problem
• Topic / token compromised or suspected leaked → rotate-credential
• Need to add a new alert that should reach ntfy → observability-integrate for the rule, then verify with § Step 4 here