AI ile A/B Test Tasarımı: Hipotezden Sonuca

A/B test = veriye dayalı karar. AI ile hipotez + analiz hızlı, istatistik anlama insan’a kalır.

Hipotez Üretimi

Rol: Sen senior CRO (conversion rate optimization) specialist.

Görev: Aşağıdaki sayfa için A/B test hipotezi:

Sayfa: SaaS pricing
Mevcut conversion (trial → paid): %2.3
Hedef: %3.5

Çıktı:
1. Top 10 hipotez:
   - Element (CTA, headline, layout, pricing display)
   - Hipotez (eğer X, then Y, çünkü Z)
   - Beklenen impact (% lift)
   - Effort (engineer / designer saat)
   - Confidence (low/med/high)
   - Risk (negative impact?)

2. Top 3 prioritize (ICE: Impact × Confidence / Effort)
3. Her birinin success metric tanımı
4. Guardrail metric (regression olmasın diye)

Klasik Test Senaryoları

CTA Button

A: "Sign Up Free" (mevcut)
B: "Start Free Trial - No Credit Card"

Hipotez: B daha çok information → conversion +%15
Sample: 5000 visitor
Duration: 2 hafta

Headline

A: "AI-Powered SEO Tool"
B: "Rank #1 on Google with AI" (benefit-focused)
C: "How 1000+ Brands Use AI for SEO" (social proof)

Multivariate: 3-arm test
Sample: 9000 visitor (3K each)

Pricing Display

A: $29/month (basit)
B: $29/month, billed annually ($348)
C: $290/year (save $58!) (annual emphasize)

Pricing display impact: %20-30 typical.

Form Fields

A: 8 fields (Name, Email, Company, Role, Industry, Size, Country, Phone)
B: 3 fields (Name, Email, Company)

Lead quality vs quantity trade-off.
Long form: less quantity, more qualified.

Social Proof

A: Just CTA
B: "Join 10,000+ users" + CTA
C: Customer logos + CTA
D: Review (5 stars) + CTA

Sample Size Hesabı

"Sample size calculator:

Mevcut:
- Conversion rate baseline: 2.3%
- Minimum detectable effect (MDE): 15% relative (= 0.345 absolute = 2.645% new rate)
- Statistical power: 80%
- Significance level: 95%

Çıktı:
- Sample per variant: X
- Total: 2X
- Test duration (günlük 1000 visitor = X gün)
- Truly negative test: erken kapatma?
"

Formula simplified:

n = (Zα/2 + Zβ)² × (p1(1-p1) + p2(1-p2)) / (p1-p2)²

%95 confidence + %80 power = (1.96 + 0.84)² ≈ 7.84

Statistical Tests

Frequentist (klasik):
- Z-test (binary outcome)
- T-test (continuous)
- Chi-square (categorical)
- ANOVA (multi-variant)

Bayesian (modern):
- Probability that B > A
- "B has 95% probability of being better"
- Continuous monitoring (peeking OK)
- Use case: business decision (not academic)

Tool default:
- Optimizely: Bayesian (Stats Engine)
- VWO: Bayesian
- Google Optimize: Bayesian
- Custom: hangi prefer

P-Value Yorumlama

"Test sonucu analiz:

Variant A: 230 conv / 10000 visitor = 2.3%
Variant B: 280 conv / 10000 visitor = 2.8%

Relative lift: +21.7%
Absolute lift: +0.5 percentage point
P-value: 0.018

Yorum:
- Statistical significance: yes (p < 0.05)
- Practical significance: 0.5 pp önemli mi business için?
- Effect size: small ama positive
- Type I error riski: %1.8 (rakip A'nın B kadar iyi olma şansı)
- Generalize: bu cohort'tan tüm population'a OK mi?
- Implementation: ship B?
"

Common Pitfalls

Peeking

❌ Her gün sonuç kontrol, %95 alır almaz durdur.
- False positive %20+ olur

✅ Pre-registered:
- Test duration before
- Sample size before
- Hypothesis before
- Sonuç sonra

Multiple Testing

10 test paralel + 95% confidence:
- En az 1 false positive %40 olasılık

Fix:
- Bonferroni: alpha / n (n = test sayısı)
- Holm correction
- FDR (False Discovery Rate) - Benjamini-Hochberg

Novelty Effect

İlk 1 hafta variant B kazanır → 4 hafta sonra eşitlenir.

Sebep: kullanıcı yeni şeye attentive.
Fix: 2-4 hafta minimum test.

Sample Ratio Mismatch (SRM)

A grup: 4500 (45%)
B grup: 5500 (55%)
Beklenen: 5000 / 5000 (50/50)

SRM = data integrity issue.
Causes:
- Bot traffic (variant cookieyok)
- JS error (B'de render fail)
- Routing bug

Fix önce, test geçersiz.

Survivorship Bias

"Only customers who completed flow" → biased.

Always include drop-off in numerator (zero conversion).

Segment Analysis

"Test sonucu segment bazlı:

Variant B overall: +15% conv

Segment:
- New user: +25%
- Returning user: -5% (!!)
- Mobile: +20%
- Desktop: +10%
- Türkiye: +30%
- Yurt dışı: 0%

Insight:
- B genelde iyi ama returning user'da kötü
- Decision: ship B ama returning user'a A göster
- Or: separate experience per segment
"

Email A/B

"Email subject line A/B:

A: "Bu hafta sızdırılan AI tool listesi"
B: "İçeride 5 AI tool var (1 saat sonra silinecek)"

Test:
- 10% audience A
- 10% audience B
- 4 saat sonra winner
- 80% audience winner (auto)

Metric: open rate
Secondary: click rate, unsubscribe

Detay: Email Marketing

Tool Setup

"PostHog A/B test setup:

Feature flag: pricing_v2
Variants:
- control (50%)
- variant_a (25%)
- variant_b (25%)

Track:
- Exposed
- Conversion (paid signup)
- Revenue per exposed

Code:
const variant = posthog.getFeatureFlag('pricing_v2');

if (variant === 'variant_a') {
  // show A
} else if (variant === 'variant_b') {
  // show B
} else {
  // control
}

Analytics:
- Daily check (peek prevention)
- Weekly review
- Stop conditions"

AI ile Variant Üretimi

"Aşağıdaki CTA için 10 variation:

Mevcut: "Sign Up Now"

Çıktı:
1. Benefit-focused: "Start Saving Time Today"
2. Specific: "Get 5 AI Tools Access"
3. Risk-removal: "Try Free, No Card"
4. Urgency: "Join 1000+ This Month"
5. Action verb: "Unlock AI Tools"
6. Personal: "Start My Free Account"
7. Exclusivity: "Get Early Access"
8. Question: "Ready to Save Hours?"
9. Soft commitment: "See How It Works"
10. Outcome: "Land Top Customers"

Her birinin:
- Hipotez (neden işe yarar)
- Use case (en uygun audience)
- Test priority"

Multi-Page Test

"Funnel optimization:

Path: Landing → Pricing → Signup → Onboarding

Test points:
- Landing: hero, CTA
- Pricing: layout, plan comparison
- Signup: form length
- Onboarding: first step

Holistic vs page-by-page:
- Holistic: end-to-end conversion (slow signal)
- Page-by-page: faster, may sub-optimize
"

Cohort Analysis

"Test variant uzun vadede:

Initial conversion B: +15%

3 ay sonra:
- Variant B kullanıcı LTV: $X
- Variant A kullanıcı LTV: $Y

B's signup quality:
- Trial-to-paid (ay 1): +15%
- Churn (ay 2): +10% (!!)
- Net LTV: -5%

Insight: 'Easier signup' = düşük kalite lead
Decision: keep A
"

Reporting

"A/B test report:

Title: Pricing Page Headline Test

Summary:
- Hipotez
- Variant
- Sonuç (winner, lift)
- Confidence
- Recommendation

Detail:
- Sample size, duration
- Methodology
- Segment breakdown
- Secondary metric (guardrail)
- Limitation
- Next test ideas

Audience: PM + Eng + Marketing
Length: 1 sayfa exec summary + 3 detay
"

Yaygın Hatalar

1. Hipotez yok: “Bir şeyi değiştir, gör”
2. Peeking + erken durdurma: False positive
3. Sample size yetersiz: Underpowered
4. Segment ignore: Aggregate yanlış conclusion
5. Novelty effect: 1 hafta yetmez
6. Multiple test düzeltme yok: False discovery
7. Long-term metric ignore: LTV → churn ignore

Sonraki Adımlar

• Veri Analizi
• SEO Anahtar Kelime
• Content Calendar
• Email Marketing
• Persona Oluşturma

Özet

A/B test + AI = hipotez 10x + variant üretimi. Sample size + significance + segment = istatistik anlama. Anahtar: peeking yapma, sample yet, segment analiz et, novelty effect dikkat. AI yardımcı, statistical understanding insan’a kalır.