benchmark models

Other Community v1.0.0

Skill notes

When to invoke this skill

Runs the same prompt through Claude, GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost, and optionally quality via LLM judge. Answers "which model is actually best for this skill?" with data instead of vibes. Separate from /benchmark, which measures web page performance. Use when: "benchmark models", "compare models",

First-run guidance (one-time)

If ACTIVATED is no (first skill run on this machine) AND the preamble printed a non-empty FIRST_TASK: value that is NOT nongit: show ONE short, project-specific line mapped from the token, as a heads-up, then CONTINUE with whatever the user actually asked — do NOT halt their task.

Voice

Direct, concrete, builder-to-builder. Name the file, function, command, and user-visible impact. No filler. No em dashes. No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted. Never corporate or academic. Short paragraphs. End with what to do. The user has context you do not. Cross-model agreement is a recommendation, not a decision. The user decides.

Completion Status Protocol

When completing a skill workflow, report status using one of: DONE — completed with evidence. DONE_WITH_CONCERNS — completed, but list concerns. BLOCKED — cannot proceed; state blocker and what was tried. NEEDS_CONTEXT — missing info; state exactly what is needed.

Operational Self-Improvement

Before completing, review the session for durable learnings and log each one — this step ALWAYS runs, it is not conditional on something feeling noteworthy (#2402: 43 of 44 learnings came from explicit /learn because "if you discovered" read as optional). A durable learning is a project quirk, command fix, pitfall, or pattern that would save 5+ minutes in a future session. If

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Other
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
92 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@garrytan · v1.0.0 · no license declared
Token usage
Heavy
Setup complexity
Manual integration
External API key
Required · Anthropic / Google
Operating systems
macOS · Linux · Windows
Runtime requirements
Bun
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview benchmark-models.preview
# Step 5: Interpret results

- **Fastest** — provider with lowest latency.
- **Cheapest** — provider with lowest cost.
- **Highest quality** (if `--judge` ran) — provider with highest score.
- **Best overall** — use judgment. If judge ran: quality-weighted. Otherwise: note the tradeoff the user needs to make.

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.