ntfy alert triage

AI Community

Skill notes

Decision tree — which question are we answering?

Symptom · Start at Got an ntfy notification, want to know what it was · § Step 1 Expected an alert, didn't get one · § Step 2 AlertmanagerClusterFailedToSendAlerts is firing · § Step 3 Want to verify the path end-to-end · § Step 4

Step 1 — What just fired?

Sort the active alerts by startsAt and find the one that matches the time the user was paged. Only alerts with rcv: containing observability/ntfy/ntfy actually went to ntfy; everything else routes to null. Ground-truth check before reacting: a fresh-looking alert can be a metric/series artifact, not a real event. For NodeUnexpectedReboot specifically, check /proc/uptime directly via talosctl; cross-reference with the cluster-triage agent memory at .Codex/agent-memory/cluster-triage/reference_reboot_alert_disambiguation.md.

Step 2 — Why didn't it deliver?

Two failure classes — distinguish by alertmanager behaviour. Class A: never reached Alertmanager — Prometheus didn't fire it, or the rule expression doesn't evaluate to truth, or the rule isn't loaded. Class B: reached Alertmanager but didn't deliver — receiver matchers excluded it, or webhook delivery failed. If your alert has severity warning/info, it routes to null by design. To reach ntfy, either bump severity in the rule or broaden the AlertmanagerConfig matcher (see ADR 0026 — "easier to broaden than to silence"; reconsider widening to "everything except info").

Step 3 — Decode the delivery failure

This is the 2026-05-05 cascade flow. When AlertmanagerClusterFailedToSendAlerts is firing, the actual error is in the alertmanager pod logs, not the alert payload. Match the error to the table: Log fragment · Root cause · Fix code":40014,"http":400,"error":"invalid request: attachments not allowed" · ntfy server has attachment-cache-dir unset; AM payload exceeds the ~5 KiB inline cap and ntfy refuses to spill · Add attachment-cache-dir: /var/cache/ntfy/attachments + size limits to kubernetes/apps/observability/ntfy/app/configmap.yaml (already fixed in main as of 2026-05-05)

Step 4 — Probe the pipeline with the ntfy CLI

ntfy is preinstalled on the operator workstation. Reference: https://docs.ntfy.sh/subscribe/cli/. The webhook URL — including the secret topic — lives in the ntfy-topic Secret in observability. Never echo the URL to stdout or write it to a file (AGENTS.md hard rule). Pipe it directly: Then split server + topic if you need them separately:

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
AI
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@wcygan · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Not required
Operating systems
Windows
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

01
Step 1 — What just fired?

Sort the active alerts by `startsAt` and find the one that matches the time the user was paged. Only alerts with `rcv:` containing `observability/ntfy/ntfy` actually went to ntfy; everything else routes to `null`. Ground-truth check before reacting: a fresh-looking alert can be a metric/series artifact, not a real event. For `NodeUnexpectedReboot` specifically, check `/proc/uptime` directly via `talosctl`; cross-reference with the cluster-triage agent memory at `.Codex/agent-memory/cluster-triage/reference_reboot_alert_disambiguation.md`.

02
Step 2 — Why didn't it deliver?

Two failure classes — distinguish by alertmanager behaviour. Class A: never reached Alertmanager — Prometheus didn't fire it, or the rule expression doesn't evaluate to truth, or the rule isn't loaded. Class B: reached Alertmanager but didn't deliver — receiver matchers excluded it, or webhook delivery failed. If your alert has `severity` warning/info, it routes to `null` by design. To reach ntfy, either bump severity in the rule or broaden the AlertmanagerConfig matcher (see ADR 0026 — "easier to broaden than to silence"; reconsider widening to "everything except info").

03
Step 3 — Decode the delivery failure

This is the 2026-05-05 cascade flow. When `AlertmanagerClusterFailedToSendAlerts` is firing, the actual error is in the alertmanager pod logs, not the alert payload. Match the error to the table: Log fragment · Root cause · Fix `code":40014,"http":400,"error":"invalid request: attachments not allowed"` · ntfy server has `attachment-cache-dir` unset; AM payload exceeds the ~5 KiB inline cap and ntfy refuses to spill · Add `attachment-cache-dir: /var/cache/ntfy/attachments` + size limits to `kubernetes/apps/observability/ntfy/app/configmap.yaml` (already fixed in main as of 2026-05-05)

04
Step 4 — Probe the pipeline with the ntfy CLI

`ntfy` is preinstalled on the operator workstation. Reference: <https://docs.ntfy.sh/subscribe/cli/>. The webhook URL — including the secret topic — lives in the `ntfy-topic` Secret in `observability`. Never echo the URL to stdout or write it to a file (AGENTS.md hard rule). Pipe it directly: Then split server + topic if you need them separately:

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.