promptfoo evaluation

Documentation Community

Skill notes

Overview

This skill provides guidance for configuring and running LLM evaluations using Promptfoo, an open-source CLI tool for testing and comparing LLM outputs.

Configuration Structure

A typical Promptfoo project structure:

Few-Shot Pattern

Embed examples directly in prompt or use chat format with assistant messages:

Python Custom Assertions

Create a Python file for custom assertions (e.g., scripts/metrics.py): Key points: Default function name is get_assert Specify function with file://path.py:function_name Return bool, float (score), or dict with pass/score/reason

LLM-as-Judge (llm-rubric)

When using a relay/proxy API, each llm-rubric assertion needs its own provider config with apiBaseUrl. Otherwise the grader falls back to the default Anthropic/OpenAI endpoint and gets 401 errors: Best practices: Provide clear scoring criteria Use threshold to set minimum passing score Default grader uses available API keys (OpenAI → Anthropic → Google)

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Documentation
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@daymade · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Required · Anthropic
Operating systems
macOS · Linux · Windows
Runtime requirements
Python
Detected file/system behavior
  • Read-only
  • Write / modify
  • Env read
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview promptfoo-evaluation.preview
# Real-World Example

tiaogaoren/
├── promptfooconfig.yaml          # Production config
├── promptfooconfig-preview.yaml  # Preview config (echo provider)
├── prompts/
│   ├── tiaogaoren-prompt.json   # Chat format with few-shot
│   └── v4/system-v4.md          # System prompt
├── tests/cases.yaml              # 3 test samples
├── scripts/metrics.py            # Custom metrics (reduction ratio, etc.)
├── data/                         # 5 samples (2 few-shot, 3 eval)
└── results/

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.