A/B Testing Platform for SaaS

Stack: Next.js 16 + Edge Middleware + GA4/GTM + Supabase + Rust-WASM (for stats) Why DIY?: Google Optimize sunset Sept 2023. GA4 has no native A/B testing.

Quick Start

// middleware.ts - Server-side assignment (no flicker)
import { NextResponse, type NextRequest } from 'next/server'

export function middleware(request: NextRequest) {
  const response = NextResponse.next()

  if (!request.cookies.get('exp_hero')) {
    const variant = Math.random() < 0.5 ? 'A' : 'B'
    response.cookies.set('exp_hero', variant, { maxAge: 60*60*24*30, path: '/' })
  }

  return response
}

// Track with GA4
window.dataLayer?.push({
  event: 'experiment_view',
  experiment_name: 'hero_test',
  experiment_variant: variant
})

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         A/B TESTING FLOW                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   1. ASSIGN (Edge Middleware)                                       │
│   ══════════════════════════                                        │
│   Request → Check cookie → Random assign → Set cookie → Response    │
│   ✓ No flicker (server-side)  ✓ Consistent (cookie-based)          │
│                                                                     │
│   2. RENDER                                                         │
│   ═════════                                                         │
│   Server/Client Component → Read cookie → Show variant              │
│                                                                     │
│   3. TRACK (GTM + GA4)                                              │
│   ════════════════════                                              │
│   dataLayer.push → GTM triggers → GA4 events with variant param     │
│                                                                     │
│   4. ANALYZE                                                        │
│   ═════════                                                         │
│   GA4 Explorations OR Supabase + Rust-WASM Bayesian analysis        │
│                                                                     │
│   5. PERSONALIZE (Advanced)                                         │
│   ═════════════════════════                                         │
│   Contextual bandit → Best variant per user segment                 │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Workflow

• 1. Define experiment config (name, variants, weight, segment rules)
• 2. Implement middleware assignment (see VARIANT-ASSIGNMENT.md)
• 3. Wire up GTM tracking (experiment_view, conversions with variant param)
• 4. Register experiment_variant as Custom Dimension in GA4
• 5. Collect data (minimum 2 weeks for weekly patterns)
• 6. Analyze significance (see STATISTICAL-ANALYSIS.md)
• 7. Roll out winner or iterate

Statistical Methods (Choose One)

Quick Bayesian (Beta-Binomial):

# A: 50/1000 conversions, B: 72/1000
import scipy.stats as stats
a_samples = stats.beta(51, 951).rvs(100000)  # Beta(1+50, 1+950)
b_samples = stats.beta(73, 929).rvs(100000)  # Beta(1+72, 1+928)
p_b_wins = (b_samples > a_samples).mean()     # → ~0.96 (96%)

Full analysis guide: STATISTICAL-ANALYSIS.md

Key Patterns

Multiple Concurrent Experiments

// middleware.ts
const EXPERIMENTS = {
  hero_cta: { weight: 0.5 },
  pricing_layout: { weight: 0.5 },
  signup_flow: { weight: 0.2 },  // 20% on new variant
}

for (const [name, config] of Object.entries(EXPERIMENTS)) {
  if (!request.cookies.get(`exp_${name}`)) {
    const variant = Math.random() < config.weight ? 'B' : 'A'
    response.cookies.set(`exp_${name}`, variant, { maxAge: 2592000, path: '/' })
  }
}

Phased Rollout

// Config stored in Supabase or Vercel Edge Config
const rolloutPhases = {
  early_access: 0.1,   // 10% new
  public_beta: 0.5,    // 50% new
  general: 1.0         // 100% new (winner)
}

Tracking Conversions

// Include variant in ALL relevant events
window.dataLayer?.push({
  event: 'sign_up',
  method: 'google',
  experiment_name: 'hero_cta',
  experiment_variant: getCookie('exp_hero_cta'),
  eventId: crypto.randomUUID()  // De-duplication
})

When to Use Rust-WASM

WASM is NOT needed for: Simple random assignment, cookie handling, event tracking

See: RUST-WASM.md

Anti-Patterns

Validation Checklist

• No flicker on page load (verify in slow 3G)
• Cookie persists across sessions (check 30-day expiry)
• GTM Preview shows correct variant in dataLayer
• GA4 DebugView receives events with variant param
• Custom dimension experiment_variant registered in GA4
• ~50/50 split verified (check GA4 Realtime)
• Conversion events include variant attribution

Reference Index

By Task

By Topic

Source: danzam98/claude-skills-toolkit — distributed by TomeVault.