警报分类
- 作者仓库星标 2
- 作者仓库 anton
- 领域
- 人工智能
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @wcygan · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- Windows
- 运行环境要求
- 未检出明确要求
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 允许外网请求
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
按 `startsAt` 对活动警报进行排序,并找到与用户被寻呼的时间匹配的警报。只有包含 `observability/ntfy/ntfy` 的 `rcv:` 警报实际上会发送至 ntfy;其他一切都路由到`null`。做出反应之前进行地面实况检查:看起来新鲜的警报可能是指标/系列工件,而不是真实事件。具体针对`NodeUnexpectedReboot`,直接通过`talosctl`查看`/proc/uptime`;与`.Codex/agent-memory/cluster-triage/reference_reboot_alert_disambiguation.md`处的集群分类代理内存交叉引用。
两种故障类别——通过警报管理器行为来区分。 A 类:从未到达 Alertmanager — Prometheus 没有触发它,或者规则表达式的计算结果不为真,或者规则未加载。 B 类:已到达 Alertmanager,但未传送 — 接收器匹配器将其排除,或者 Webhook 传送失败。如果您的警报有`severity`警告/信息,它会根据设计路由到`null`。要达到 ntfy,请提高规则的严重性或扩大 AlertmanagerConfig 匹配器(请参阅 ADR 0026 —“扩大比沉默更容易”;重新考虑扩大到“除信息之外的所有内容”)。
这是2026-05-05的级联流程。当 `AlertmanagerClusterFailedToSendAlerts` 触发时,实际错误位于 Alertmanager Pod 日志中,而不是警报有效负载中。将错误与表进行匹配:日志片段 · 根本原因 · 修复`code":40014,"http":400,"error":"invalid request: attachments not allowed"` · ntfy 服务器未设置`attachment-cache-dir`; AM 有效负载超过 ~5 KiB 内联上限,并且 ntfy 拒绝溢出 · 将 `attachment-cache-dir: /var/cache/ntfy/attachments` + 大小限制添加到 `kubernetes/apps/observability/ntfy/app/configmap.yaml`(截至 2026 年 5 月 5 日已在 main 中修复)
`ntfy` 预装在操作员工作站上。参考:<https://docs.ntfy.sh/subscribe/cli/>。 Webhook URL(包括秘密主题)位于 `observability` 中的 `ntfy-topic` Secret 中。切勿将 URL 回显到标准输出或将其写入文件(AGENTS.md 硬规则)。直接通过管道传输:如果您单独需要服务器+主题,则将它们分开:
# ntfy alert triage
Ordered triage for the **anton** alert pipeline: PrometheusRule → Alertmanager → AlertmanagerConfig route → webhook → self-hosted ntfy at `ntfy.<tailnet>.ts.net` → (optionally) ntfy.sh upstream relay → device. The 2026-05-05 cascade where a 40014 from ntfy turned into a 5-minute `AlertmanagerClusterFailedToSendAlerts` loop is the canonical failure mode this skill is built around.
## Pipeline map (memorise this)
```
PrometheusRule Helm-managed via kube-prometheus-stack;
│ Anton-authored rules live in
│ kubernetes/apps/observability/kube-prometheus-stack/app/
▼
Alertmanager (alertmanager-kube-prometheus-stack-alertmanager-0)
│ route: severity=critical → observability/ntfy/ntfy receiver
│ everything else → "null" receiver
│ (see AlertmanagerConfig at kubernetes/apps/observability/ntfy/app/alertmanagerconfig.yaml)
▼
webhook POST to URL from Secret ntfy-topic in observability ns
│ (templated by ESO from 1Password ntfy/topic)
▼
self-hosted ntfy (Deployment ntfy in observability)
│ base-url: https://ntfy.<tailnet>.ts.net
│ upstream-base-url: https://ntfy.sh ← iOS push relay (ADR 0026)
│ attachment-cache-dir set since 2026-05-05 (configmap.yaml)
▼
device (browser at https://ntfy.<tailnet>/<topic>, or iOS via ntfy.sh poll trigger)
```
## Decision tree — which question are we answering?
| Symptom | Start at |
|---|---|
| Got an ntfy notification, want to know what it was | [§ Step 1](#step-1--what-just-fired) |
| Expected an alert, didn't get one | [§ Step 2](#step-2--why-didnt-it-deliver) |
| `AlertmanagerClusterFailedToSendAlerts` is firing | [§ Step 3](#step-3--decode-the-delivery-failure) |
| Want to verify the path end-to-end | [§ Step 4](#step-4--probe-the-pipeline-with-the-ntfy-cli) |
| Need to write/tighten a rule or receiver | hand off to `observability-integrate` |
---
## Step 1 — What just fired?
```sh
kubectl exec -n observability alertmanager-kube-prometheus-stack-alertmanager-0 \
-c alertmanager -- wget -qO- \
'http://localhost:9093/api/v2/alerts?active=true&silenced=false&inhibited=false' \
| python3 -c "import json,sys; [print(a['labels'].get('alertname'),'|',a['labels'].get('severity'),'|',a['labels'].get('instance',''),'|','start:',a['startsAt'],'|','rcv:',[r['name'] for r in a['receivers']]) for a in json.load(sys.stdin)]"
```
Sort the active alerts by `startsAt` and find the one that matches the time the user was paged. Only alerts with `rcv:` containing `observability/ntfy/ntfy` actually went to ntfy; everything else routes to `null`.
**Ground-truth check before reacting**: a fresh-looking alert can be a metric/series artifact, not a real event. For `NodeUnexpectedReboot` specifically, check `/proc/uptime` directly via `talosctl`; cross-reference with the cluster-triage agent memory at `.Codex/agent-memory/cluster-triage/reference_reboot_alert_disambiguation.md`.
---
## Step 2 — Why didn't it deliver?
Two failure classes — distinguish by alertmanager behaviour.
**Class A: never reached Alertmanager** — Prometheus didn't fire it, or the rule expression doesn't evaluate to truth, or the rule isn't loaded.
```sh
# Did Prometheus load the rule?
kubectl exec -n observability prometheus-kube-prometheus-stack-prometheus-0 \
-c prometheus -- wget -qO- http://localhost:9090/api/v1/rules \
| python3 -c "import json,sys; [print(g['name'],'/',r['name']) for f in json.load(sys.stdin)['data']['groups'] for r in f['rules'] for g in [f]]" \
| grep -i <alertname>
# Is the expression returning samples right now?
# (Pull the expr from the PrometheusRule yaml and test it via /api/v1/query.)
```
**Class B: reached Alertmanager but didn't deliver** — receiver matchers excluded it, or webhook delivery failed.
```sh
# Receiver matchers — only severity=critical reaches ntfy in anton.
kubectl exec -n observability alertmanager-kube-prometheus-stack-alertmanager-0 \
-c alertmanager -- wget -qO- http://localhost:9093/api/v2/status \
| python3 -c "import json,sys,re; print(re.search(r'route:.*?inhibit_rules', json.load(sys.stdin)['config']['original'], re.S).group(0))"
```
If your alert has `severity` warning/info, it routes to `null` by design. To reach ntfy, either bump severity in the rule or broaden the AlertmanagerConfig matcher (see ADR 0026 — "easier to broaden than to silence"; reconsider widening to "everything except info").
---
## Step 3 — Decode the delivery failure
This is the 2026-05-05 cascade flow. When `AlertmanagerClusterFailedToSendAlerts` is firing, the actual error is in the alertmanager pod logs, not the alert payload.
```sh
kubectl logs -n observability alertmanager-kube-prometheus-stack-alertmanager-0 \
-c alertmanager --tail=200 | grep -iE 'error|fail|notify' | tail -20
```
Match the error to the table:
| Log fragment | Root cause | Fix |
|---|---|---|
| `code":40014,"http":400,"error":"invalid request: attachments not allowed"` | ntfy server has `attachment-cache-dir` unset; AM payload exceeds the ~5 KiB inline cap and ntfy refuses to spill | Add `attachment-cache-dir: /var/cache/ntfy/attachments` + size limits to `kubernetes/apps/observability/ntfy/app/configmap.yaml` (already fixed in main as of 2026-05-05) |
| `unexpected status code 401` | ntfy ACL added without updating the webhook URL secret | Refresh the `ntfy-topic` secret (1Password `ntfy/topic`) — see `rotate-credential` |
| `dial tcp ... no route to host` / `connection refused` | ntfy pod down or service IP changed | `kubectl get pod,svc -n observability -l app.kubernetes.io/name=ntfy` |
| `x509: certificate signed by unknown authority` | TLS chain regression on the cluster_gateway / cert-manager | Hand off to `anton-cluster-health` layer-5 |
| `context deadline exceeded` | ntfy.sh upstream slow; usually transient | Wait one repeat-interval; if persistent, check status.ntfy.sh |
| repeated `notify retry canceled due to unrecoverable error` for the same alert group | ntfy returning 4xx — the alert payload itself is malformed for the receiver | Decode the request body shape (see [§ Step 4](#step-4--probe-the-pipeline-with-the-ntfy-cli)) |
**Inhibit while fixing**: the cluster-failed-to-send alert will keep paging while you work. Silence it via `amtool silence add alertname=AlertmanagerClusterFailedToSendAlerts --duration=30m -c <annotation>`, or accept ~5-15 min of stale alerts after the fix lands.
---
## Step 4 — Probe the pipeline with the ntfy CLI
`ntfy` is preinstalled on the operator workstation. Reference: <https://docs.ntfy.sh/subscribe/cli/>.
The webhook URL — including the secret topic — lives in the `ntfy-topic` Secret in `observability`. **Never echo the URL to stdout or write it to a file** (AGENTS.md hard rule). Pipe it directly:
```sh
# Read the URL into a shell variable WITHOUT printing it.
# (Single command; topic stays out of shell history if HISTCONTROL=ignorespace and you prefix with a space.)
NTFY_URL=$(kubectl get secret -n observability ntfy-topic -o jsonpath='{.data.url}' | base64 -d)
```
Then split server + topic if you need them separately:
```sh
NTFY_SERVER="${NTFY_URL%/*}"
NTFY_TOPIC="${NTFY_URL##*/}"
```
### Read recent deliveries (no long-running connection)
```sh
ntfy subscribe --poll --since 30m "${NTFY_URL}"
```
Use `--since 1h` / `--since 12h` for wider windows (ntfy `cache-duration: 12h` in anton's ConfigMap; older messages are gone). The `--poll` flag fetches and exits — never start a backgrounded `ntfy subscribe` from this skill.
### Send a test publish
```sh
ntfy publish --title "anton triage probe" --priority default --tags test \
"${NTFY_URL}" "$(date -u +%FT%TZ) probe from ntfy-alert-triage skill"
```
A successful publish prints the message ID and returns 0. A 40014 here means attachments are still disabled (see Step 3). A 401 means ACLs were added without updating the secret. A 404 on the topic means the topic name doesn't match the URL path.
### Reproduce the 40014 (size-spill check)
The original cascade was triggered by Alertmanager grouping 2+ alerts into one webhook body that exceeded ~5 KiB. To verify attachments-cache works end-to-end:
```sh
ntfy publish "${NTFY_URL}" "$(python3 -c 'print("x"*8000)')"
```
Pre-fix this returned 40014; post-fix it succeeds and the message is stored as an attachment (visible in the ntfy web UI, not inline on iOS).
### Verify the iOS upstream relay
iOS push goes via `upstream-base-url: https://ntfy.sh` (ADR 0026 — only message IDs transit; bodies stay on cluster). To check the upstream poll registration:
```sh
kubectl logs -n observability deploy/ntfy --tail=50 | grep -i upstream
```
A healthy line looks like `Successfully forwarded message to upstream`. Errors mean iOS won't get push notifications even when desktop browser delivery works.
---
## Anton-specific reference
| Where it lives | What it is |
|---|---|
| `kubernetes/apps/observability/ntfy/app/alertmanagerconfig.yaml` | The route — currently `severity=critical` only (per ADR 0026 "easier to broaden than silence") |
| `kubernetes/apps/observability/ntfy/app/configmap.yaml` | ntfy server.yml — base-url, upstream-base-url, attachment limits |
| `kubernetes/apps/observability/ntfy/app/externalsecret.yaml` | ESO mapping that fills the `ntfy-topic` Secret from 1Password |
| `kubernetes/apps/observability/ntfy/app/deployment.yaml` | Single-replica ntfy v2.x, RWO cache PVC, Recreate strategy |
| ADR 0026 (`context/adrs/0026-self-hosted-ntfy-as-alertmanager-destination.md`) | Why ntfy, why upstream relay, broadening policy |
| Postmortem 2026-05-05 (alert cascade) | Captured as commit `084babaa` body — three coordinated fixes |
## Hard rules (carry-overs from AGENTS.md)
- **Never** echo, log, or write the topic URL or secret value. Always pipe `kubectl get secret ... -o jsonpath` directly into the consuming command.
- **Never** edit `*.sops.*` files in plaintext; use `sops <file>` for round-trip.
- **Never** restart the alertmanager StatefulSet to "fix" a delivery problem — the failure is upstream of AM in 99% of cases. Restart ntfy if the ntfy pod is the suspect; restart AM only if its own logs say so.
- The Vector kernel sink (`talos-log-sink`) and its 30Gi PVC are unrelated to ntfy alerting; if both are firing alerts, treat them independently.
## Hand-offs
- Rule expression is wrong / missing → `observability-integrate`
- ntfy pod itself is broken (CrashLoop, OOM, image pull) → `anton-cluster-health` layer 5, then `debug-flux-reconciliation` if it's a Flux apply problem
- Topic / token compromised or suspected leaked → `rotate-credential`
- Need to add a new alert that should reach ntfy → `observability-integrate` for the rule, then verify with [§ Step 4](#step-4--probe-the-pipeline-with-the-ntfy-cli) here
按说明文件中的顺序排列。
决策树——我们要回答哪个问题?
第 1 步——刚刚发射了什么?
第 2 步 — 为什么没有交付?
第 3 步 — 解码交付失败
步骤 4 — 使用 ntfy CLI 探测管道
文件/命令 -> ntfy.<tailnet>.ts.net · AlertmanagerClusterFailedToSendAlerts · observability-integrate · startsAt · rcv: · observability/ntfy/ntfy · null · NodeUnexpectedReboot
内容校验值 -> 90cf6c0962b1
当前原文的自动摘译,尚未人工审校。
决策树——我们要回答哪个问题?
症状 · 开始于 收到 ntfy 通知,想知道它是什么 · § 步骤 1 预期有警报,但没有收到 · § 步骤 2
AlertmanagerClusterFailedToSendAlerts正在触发 · § 步骤 3 想要端到端验证路径 · § 步骤 4第 1 步——刚刚发射了什么?
按
startsAt对活动警报进行排序,并找到与用户被寻呼的时间相匹配的警报。只有包含observability/ntfy/ntfy的rcv:警报实际上会发送至 ntfy;其他一切都路由到null。做出反应之前进行地面实况检查:看起来新鲜的警报可能是指标/系列工件,而不是真实事件。具体针对NodeUnexpectedReboot,直接通过talosctl查看/proc/uptime;与.Codex/agent-memory/cluster-triage/reference_reboot_alert_disambiguation.md处的集群分类代理内存进行交叉引用。第 2 步 — 为什么没有交付?
两种故障类别——通过警报管理器行为来区分。 A 类:从未到达 Alertmanager — Prometheus 没有触发它,或者规则表达式的计算结果不为真,或者规则未加载。 B 类:已到达 Alertmanager,但未传送 — 接收器匹配器将其排除,或者 Webhook 传送失败。如果您的警报有
severity警告/信息,它会按照设计路由到null。要达到 ntfy,请提高规则的严重性或扩大 AlertmanagerConfig 匹配器(请参阅 ADR 0026 —“扩大比沉默更容易”;重新考虑扩大到“除信息之外的所有内容”)。第 3 步 — 解码交付失败
这是2026-05-05的级联流程。当
AlertmanagerClusterFailedToSendAlerts触发时,实际错误位于 Alertmanager Pod 日志中,而不是警报负载中。将错误与表进行匹配:日志片段 · 根本原因 · 修复code":40014,"http":400,"error":"invalid request: attachments not allowed"· ntfy 服务器未设置attachment-cache-dir; AM 有效负载超过 ~5 KiB 内联上限,ntfy 拒绝溢出 · 将attachment-cache-dir: /var/cache/ntfy/attachments+ 大小限制添加到kubernetes/apps/observability/ntfy/app/configmap.yaml(截至 2026 年 5 月 5 日已在 main 中修复)