ab testing
- Repo stars 0
- Author repo skills-registry
- Domain
- Engineering
- Author-declared agents
- No explicit declaration found; this is not inferred or tested compatibility
- Static check
- 88 / 100 · heuristic scan, not runtime safety proof
- Runtime evidence
- Not run; no isolated runtime test evidence is currently recorded
- Author / version / license
- @tomevault-io · no license declared
- Token usage
- Lean
- Setup complexity
- Plug-and-play
- External API key
- Not required
- Operating systems
- Unspecified (assume cross-platform)
- Runtime requirements
- Python
- Detected file/system behavior
-
- Read-only
- Write / modify
- Network behavior
- Local-only
- Install commands
- None (reference only)
Heads up: Tool permissions are not declared; actual access depends on the runtime.
# A/B Testing Platform for SaaS
> **Stack**: Next.js 16 + Edge Middleware + GA4/GTM + Supabase + Rust-WASM (for stats)
> **Why DIY?**: Google Optimize sunset Sept 2023. GA4 has no native A/B testing.
## Quick Start
```tsx
// middleware.ts - Server-side assignment (no flicker)
import { NextResponse, type NextRequest } from 'next/server'
export function middleware(request: NextRequest) {
const response = NextResponse.next()
if (!request.cookies.get('exp_hero')) {
const variant = Math.random() < 0.5 ? 'A' : 'B'
response.cookies.set('exp_hero', variant, { maxAge: 60*60*24*30, path: '/' })
}
return response
}
```
```tsx
// Track with GA4
window.dataLayer?.push({
event: 'experiment_view',
experiment_name: 'hero_test',
experiment_variant: variant
})
```
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ A/B TESTING FLOW │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ 1. ASSIGN (Edge Middleware) │
│ ══════════════════════════ │
│ Request → Check cookie → Random assign → Set cookie → Response │
│ ✓ No flicker (server-side) ✓ Consistent (cookie-based) │
│ │
│ 2. RENDER │
│ ═════════ │
│ Server/Client Component → Read cookie → Show variant │
│ │
│ 3. TRACK (GTM + GA4) │
│ ════════════════════ │
│ dataLayer.push → GTM triggers → GA4 events with variant param │
│ │
│ 4. ANALYZE │
│ ═════════ │
│ GA4 Explorations OR Supabase + Rust-WASM Bayesian analysis │
│ │
│ 5. PERSONALIZE (Advanced) │
│ ═════════════════════════ │
│ Contextual bandit → Best variant per user segment │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
---
## Workflow
- [ ] 1. Define experiment config (name, variants, weight, segment rules)
- [ ] 2. Implement middleware assignment (see [VARIANT-ASSIGNMENT.md](references/VARIANT-ASSIGNMENT.md))
- [ ] 3. Wire up GTM tracking (experiment_view, conversions with variant param)
- [ ] 4. Register `experiment_variant` as Custom Dimension in GA4
- [ ] 5. Collect data (minimum 2 weeks for weekly patterns)
- [ ] 6. Analyze significance (see [STATISTICAL-ANALYSIS.md](references/STATISTICAL-ANALYSIS.md))
- [ ] 7. Roll out winner or iterate
---
## Statistical Methods (Choose One)
| Method | Best For | Decision Output |
|--------|----------|-----------------|
| **Frequentist** | Fixed sample, strict control | p-value < 0.05 → significant |
| **Bayesian** | Continuous monitoring, intuitive | P(B > A) = 96% → B likely better |
| **Multi-Armed Bandit** | Optimize during test | Auto-shift traffic to winner |
| **Contextual Bandit** | Personalization | Best variant per user segment |
**Quick Bayesian (Beta-Binomial)**:
```python
# A: 50/1000 conversions, B: 72/1000
import scipy.stats as stats
a_samples = stats.beta(51, 951).rvs(100000) # Beta(1+50, 1+950)
b_samples = stats.beta(73, 929).rvs(100000) # Beta(1+72, 1+928)
p_b_wins = (b_samples > a_samples).mean() # → ~0.96 (96%)
```
**Full analysis guide**: [STATISTICAL-ANALYSIS.md](references/STATISTICAL-ANALYSIS.md)
---
## Key Patterns
### Multiple Concurrent Experiments
```tsx
// middleware.ts
const EXPERIMENTS = {
hero_cta: { weight: 0.5 },
pricing_layout: { weight: 0.5 },
signup_flow: { weight: 0.2 }, // 20% on new variant
}
for (const [name, config] of Object.entries(EXPERIMENTS)) {
if (!request.cookies.get(`exp_${name}`)) {
const variant = Math.random() < config.weight ? 'B' : 'A'
response.cookies.set(`exp_${name}`, variant, { maxAge: 2592000, path: '/' })
}
}
```
### Phased Rollout
```tsx
// Config stored in Supabase or Vercel Edge Config
const rolloutPhases = {
early_access: 0.1, // 10% new
public_beta: 0.5, // 50% new
general: 1.0 // 100% new (winner)
}
```
### Tracking Conversions
```tsx
// Include variant in ALL relevant events
window.dataLayer?.push({
event: 'sign_up',
method: 'google',
experiment_name: 'hero_cta',
experiment_variant: getCookie('exp_hero_cta'),
eventId: crypto.randomUUID() // De-duplication
})
```
---
## When to Use Rust-WASM
| Use Case | Why Rust |
|----------|----------|
| Monte Carlo simulation (100k+ draws) | 10-100x faster than JS |
| Bayesian posterior computation | Numerical precision |
| Contextual bandit inference | Real-time ML at edge |
| Cross-platform consistency | Same logic in browser + server |
**WASM is NOT needed for**: Simple random assignment, cookie handling, event tracking
See: [RUST-WASM.md](references/RUST-WASM.md)
---
## Anti-Patterns
| Don't | Why |
|-------|-----|
| Client-side variant assignment | Causes flicker, inconsistent |
| End test early ("B winning after 2 days!") | Random noise, not signal |
| Multiple changes in one variant | Can't isolate what worked |
| Overlapping tests on same element | Interaction effects confound |
| Skip sample size calculation | Under-powered = false negatives |
| Ignore segments | Winner overall may lose for key segment |
---
## Validation Checklist
- [ ] No flicker on page load (verify in slow 3G)
- [ ] Cookie persists across sessions (check 30-day expiry)
- [ ] GTM Preview shows correct variant in dataLayer
- [ ] GA4 DebugView receives events with variant param
- [ ] Custom dimension `experiment_variant` registered in GA4
- [ ] ~50/50 split verified (check GA4 Realtime)
- [ ] Conversion events include variant attribution
---
## Reference Index
### By Task
| I need to... | Read |
|--------------|------|
| **Implement variant assignment** | [VARIANT-ASSIGNMENT.md](references/VARIANT-ASSIGNMENT.md) |
| **Choose a statistical method** | [STATISTICAL-ANALYSIS.md](references/STATISTICAL-ANALYSIS.md) |
| **Set up GTM/GA4 tracking** | [GA4-GTM-TRACKING.md](references/GA4-GTM-TRACKING.md) |
| **Build admin dashboard** | [ADMIN-DASHBOARD.md](references/ADMIN-DASHBOARD.md) |
| **Add personalization/bandits** | [PERSONALIZATION.md](references/PERSONALIZATION.md) |
| **Optimize with Rust-WASM** | [RUST-WASM.md](references/RUST-WASM.md) |
| **Quick lookup (tools, formulas)** | [QUICK-REFERENCE.md](references/QUICK-REFERENCE.md) |
### By Topic
| Topic | Reference |
|-------|-----------|
| Server-side assignment, multiple experiments, weighted splits | [VARIANT-ASSIGNMENT.md](references/VARIANT-ASSIGNMENT.md) |
| Frequentist vs Bayesian vs Bandits, sample size, pitfalls | [STATISTICAL-ANALYSIS.md](references/STATISTICAL-ANALYSIS.md) |
| GTM variables, GA4 events, BigQuery queries, debugging | [GA4-GTM-TRACKING.md](references/GA4-GTM-TRACKING.md) |
| Database schema, API routes, UI components, real-time updates | [ADMIN-DASHBOARD.md](references/ADMIN-DASHBOARD.md) |
| Segments, rules-based, Thompson sampling, contextual bandits | [PERSONALIZATION.md](references/PERSONALIZATION.md) |
| Rust setup, beta sampling, bandit implementation, Next.js integration | [RUST-WASM.md](references/RUST-WASM.md) |
| Tool recommendations, decision guide, formulas, cheat sheet | [QUICK-REFERENCE.md](references/QUICK-REFERENCE.md) |
---
> Source: [danzam98/claude-skills-toolkit](https://github.com/danzam98/claude-skills-toolkit) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-05-23 -->
Sections and files
Listed in source order.
sections -> Quick Start → Architecture → Workflow → Statistical Methods (Choose One) → Key Patterns → Multiple Concurrent Experiments
terms -> Stack · Why DIY? · Frequentist · Bayesian · Multi-Armed Bandit · Contextual Bandit · Quick Bayesian (Beta-Binomial) · Full analysis guide
files/cmd -> experiment_variant · exp_${name} · A/B · Next.js · GA4/GTM · middleware.ts · next/server · Server/Client
body sha256 -> 7512d8f6e928
Workflow
[ ] 1. Define experiment config (name, variants, weight, segment rules) [ ] 2. Implement middleware assignment (see VARIANT-ASSIGNMENT.md) [ ] 3. Wire up GTM tracking (experiment_view, conversions with variant param) [ ] 4. Register
experiment_variantas Custom Dimension in GA4 [ ] 5. Collect data (minimum 2 weeks for weekly patterns)Statistical Methods (Choose One)
Method · Best For · Decision Output Frequentist · Fixed sample, strict control · p-value < 0.05 → significant Bayesian · Continuous monitoring, intuitive · P(B > A) = 96% → B likely better Multi-Armed Bandit · Optimize during test · Auto-shift traffic to winner Contextual Bandit · Personalization · Best variant per user segment
When to Use Rust-WASM
Use Case · Why Rust Monte Carlo simulation (100k+ draws) · 10-100x faster than JS Bayesian posterior computation · Numerical precision Contextual bandit inference · Real-time ML at edge Cross-platform consistency · Same logic in browser + server
Anti-Patterns
Don't · Why Client-side variant assignment · Causes flicker, inconsistent End test early ("B winning after 2 days!") · Random noise, not signal Multiple changes in one variant · Can't isolate what worked Overlapping tests on same element · Interaction effects confound
Validation Checklist
[ ] No flicker on page load (verify in slow 3G) [ ] Cookie persists across sessions (check 30-day expiry) [ ] GTM Preview shows correct variant in dataLayer [ ] GA4 DebugView receives events with variant param [ ] Custom dimension
experiment_variantregistered in GA4