vector forge

AI Community

Skill notes

When to Use

Generating test vectors for cryptographic algorithms or protocols Evaluating how well existing test vectors cover an implementation Finding implementation code paths that no test vector exercises Creating Wycheproof-style cross-implementation test vectors Measuring the concrete coverage value of a test vector suite

When NOT to Use

No implementations exist yet (need code to mutate) Single trivial implementation with no edge cases Testing application logic rather than algorithm implementations The algorithm has no public test vectors to compare against

Prerequisites

trailmark installed — if uv run trailmark fails, run: At least one implementation of the target algorithm in a language with mutation testing support A test harness that consumes test vectors and exercises the implementation

Rationalizations to Reject

Rationalization · Why It's Wrong · Required Action "We have enough test vectors" · Mutation testing proves otherwise · Run the baseline first "The implementation's own tests are sufficient" · Own tests often share blind spots with the impl · Cross-impl vectors catch different bugs "FFI crates can be mutation tested at the binding layer" · Mutations to wrappers don't affect the underlying impl · Mutate the actual implementation language

Phase 1: Discovery

Find implementations of the target algorithm. Look for: Pure implementations in high-level languages (Go, Rust, Python) — these are the best mutation testing targets FFI wrapper crates — identify these early so you don't waste time mutating wrapper glue code

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
AI
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@trailofbits · no license declared
Token usage
Moderate
Setup complexity
Guided setup
External API key
Not required
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
Python
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview vector-forge.preview
# Recording Baseline Results

Capture these metrics per implementation: | Metric | Description | |--------|-------------| | Total mutants | Number of mutations generated | | Killed | Mutants caught by tests | | Survived/Lived | Mutants NOT caught (these are the targets) | | Not covered | Code paths no test reaches at all | | Timed out | Ambiguous — resolve before comparing | | Efficacy % | Killed / (Killed + Survived) | | Coverage % | (Total - Not covered) / Total | Save the full mutation log for Phase 4 analysis. ---

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.