A/B Testing & Experimentation Analysis (R)

Overview

This project builds a 7-script experimentation pipeline in R simulating a SaaS A/B test comparing a new onboarding flow (treatment) against the current experience (control) across 10,000 users. The pipeline covers every stage of a rigorous experiment analysis – from data generation through power assessment.

The analysis follows industry methodology used at companies like Microsoft, Booking.com, and Netflix: verify randomization integrity (SRM) first, then analyze the primary metric, check guardrail metrics, validate with distribution-free inference (bootstrap/permutation), refine estimates with covariate adjustment, and assess statistical power for future experiment design.

Business Context

A/B testing is the gold standard for measuring the causal impact of product changes. This project simulates a realistic scenario: a SaaS company redesigns its onboarding flow, expecting to lift conversion rates and reduce time-to-complete. The pipeline demonstrates how to move from raw experiment data to a defensible ship/no-ship decision with quantified uncertainty.

Objectives

• Generate a realistic simulated dataset with known ground-truth effects for pipeline validation
• Verify randomization integrity via Sample Ratio Mismatch (SRM) and covariate balance tests
• Measure the primary conversion metric with confidence intervals and effect sizes
• Check guardrail metrics (time-to-complete) to ensure no unintended regressions
• Validate results with bootstrap CIs and permutation tests (distribution-free inference)
• Refine treatment effect estimates using regression adjustment with robust standard errors
• Calculate minimum detectable effect (MDE) and required sample sizes for future experiments

Dataset Overview

Tools & Packages

Key Metrics Defined

Approach

Using simulated data with a known ground truth is the most rigorous way to validate an analysis pipeline. If the analysis recovers the planted 1.5pp conversion lift and 0.5-minute time reduction, the methodology is confirmed to work correctly. This is standard practice in experimentation platform development – teams at Microsoft and Netflix use simulation to verify their analysis code before deploying it on real experiments.

The simulation generates 10,000 users with 50/50 random assignment to control or treatment. Control baseline conversion is 10%, treatment adds a 1.5 percentage-point lift (11.5% true rate). Time-to-complete is normally distributed – control centered at 8.0 minutes, treatment at 7.5 minutes (0.5 min faster), both with SD=3. A pre-experiment covariate (pre_sessions_7d, Poisson λ=3) is included to enable regression adjustment (CUPED) in later scripts.

Key Code

# Simulation parameters
n_users <- 10000
baseline_conversion <- 0.10
treatment_lift <- 0.015  # +1.5 percentage points

df <- tibble(
  user_id = 1:n_users,
  variant = sample(c("control", "treatment"), size = n_users, replace = TRUE),
  pre_sessions_7d = rpois(n_users, lambda = 3)
) %>%
  mutate(
    conversion_prob = case_when(
      variant == "control"   ~ baseline_conversion,
      variant == "treatment" ~ baseline_conversion + treatment_lift
    ),
    converted = rbinom(n_users, 1, conversion_prob),
    time_to_complete = rnorm(n_users,
      mean = ifelse(variant == "treatment", 7.5, 8.0), sd = 3)
  )

Key Points

• Reproducible via set.seed(123) – every run produces identical data
• Ground truth is known: +1.5pp conversion lift, −0.5 min time-to-complete
• Pre-experiment covariate (pre_sessions_7d) included for regression adjustment
• Clean CSV output (data/ab_test_data.csv) feeds all downstream scripts

Approach

Before analyzing any experiment results, the first step is verifying randomization integrity. Sample Ratio Mismatch (SRM) is the single most critical quality check – if the observed 50/50 split deviates significantly from expectation, something is wrong with assignment, logging, or data processing. SRM detection is standard practice at Microsoft, Booking.com, and other companies running experiments at scale.

The pipeline applies a chi-square goodness-of-fit test against the expected 50/50 proportions and a Welch two-sample t-test to verify that the pre-experiment covariate (pre_sessions_7d) is balanced across variants. If SRM is detected (p < 0.01), the pipeline halts – analyzing results from a compromised randomization is worse than analyzing no results at all.

Key Code

# SRM test (expected 50/50 split)
counts <- df %>% count(variant)
chisq <- chisq.test(x = counts$n, p = c(0.5, 0.5))

# Baseline covariate balance
tt <- t.test(pre_sessions_7d ~ variant, data = df)

# Fail loudly if SRM detected
if (chisq$p.value < 0.01) {
  stop("SRM detected – investigate randomization before proceeding.")
}

Results

Group Counts

SRM Test Result

Baseline Covariate Balance

Welch t-test for covariate balance: p = 0.188

Key Findings

• No SRM detected (chi-square p = 0.734, well above the 0.01 threshold) – randomization is intact
• Baseline covariate pre_sessions_7d is balanced across variants (t-test p = 0.188) – no pre-existing group differences
• Pipeline cleared to proceed with primary and guardrail metric analysis

Business Recommendations

• Always run SRM before analyzing any experiment – it is the single most important integrity check
• Automate SRM checks into experiment platforms to catch issues before analysts examine results
• If SRM is detected, do NOT analyze results – investigate the root cause (logging bugs, bot traffic, assignment errors) first

Approach

Conversion rate is the primary decision metric for this experiment. The analysis uses a two-proportion z-test (prop.test) comparing control vs. treatment conversion rates. This is the standard frequentist approach for binary outcome experiments.

The analysis reports absolute lift (percentage points), relative lift (%), a 95% confidence interval for the difference, and Cohen's h as a standardized effect size. The distinction between statistical significance and practical significance matters – the width of the confidence interval tells us about the range of plausible effects, which is more informative than a binary p-value threshold.

Key Code

# Two-proportion z-test
pt <- prop.test(
  x = c(x_control, x_treat),
  n = c(n_control, n_treat),
  correct = FALSE
)

# Effect size (Cohen's h)
cohens_h <- 2 * asin(sqrt(p_treat)) - 2 * asin(sqrt(p_control))

# Absolute and relative lift
abs_lift <- p_treat - p_control
rel_lift <- (p_treat / p_control) - 1

Results

Conversion Summary

Test Results

Key Findings

• Treatment conversion rate (11.52%) is 1.33pp higher than control (10.19%) – close to the planted 1.5pp effect
• The 95% confidence interval [0.11pp, 2.55pp] excludes zero, confirming statistical significance at α=0.05
• Relative lift of 13.1% over a 10% baseline – a meaningful improvement for SaaS onboarding
• Cohen's h = 0.043 indicates a small but real effect size, consistent with typical product A/B tests

Business Recommendations

• A 1.33pp lift on a 10% baseline represents a ~13% relative improvement – meaningful for SaaS onboarding funnels where even small gains compound across thousands of users
• Always report confidence intervals alongside p-values – the range of plausible effects (0.11pp to 2.55pp) is more useful than binary significance
• Estimate business impact: at 100K annual signups, a 1.33pp lift translates to ~1,330 additional conversions per year

Approach

Guardrail metrics ensure the treatment does not cause unintended harm. Even if conversion improves, if the onboarding process takes significantly longer, the user experience may suffer – leading to downstream churn or support costs that offset the conversion gain.

The analysis compares time-to-complete using three complementary methods: a Welch t-test (parametric), Wilcoxon rank-sum test (non-parametric), and bootstrap CI (distribution-free). Using multiple approaches validates robustness, especially when the underlying distribution may be skewed or heavy-tailed.

Key Code

# Parametric + nonparametric tests
tt <- t.test(time_to_complete ~ variant, data = df)
wt <- wilcox.test(time_to_complete ~ variant, data = df, exact = FALSE)

# Bootstrap CI for mean difference
boot_diffs <- replicate(2000, {
  boot_df <- df %>% group_by(variant) %>%
    slice_sample(prop = 1, replace = TRUE)
  mean(boot_df$time_to_complete[boot_df$variant == "treatment"]) -
    mean(boot_df$time_to_complete[boot_df$variant == "control"])
})
boot_ci <- quantile(boot_diffs, probs = c(0.025, 0.975))

Visualization

Results

Time-to-Complete Summary

Test Results

Key Findings

• Treatment group completed onboarding 0.34 minutes faster on average – consistent with the planted 0.5-min effect (sampling variability accounts for the difference)
• All three testing approaches (t-test, Wilcoxon, bootstrap) agree: the difference is highly significant (p < 0.001)
• Bootstrap CI [−0.46, −0.22] is entirely below zero – the guardrail metric shows improvement, not regression
• The guardrail is clear: the new onboarding flow is both more effective (higher conversion) and faster (lower time-to-complete)

Business Recommendations

• Even a 0.34-minute reduction compounds across thousands of users – calculate total time saved per month to quantify UX improvement
• Use multiple testing approaches when the metric distribution may be non-normal – agreement across methods strengthens confidence
• Always check guardrail metrics before shipping – a conversion lift that degrades the user experience may not be worth it

Approach

This section validates the primary conversion results using computation-based inference that does not rely on distributional assumptions. These methods are increasingly preferred in industry experimentation because they provide intuitive uncertainty quantification without requiring normality or large-sample approximations.

Bootstrap confidence intervals (B=4,000 resamples) resample within each group to build a distribution of the observed lift. This is the gold standard for uncertainty quantification in modern experimentation platforms – it directly answers "what range of lifts is consistent with the data?"

The permutation test (B=4,000 shuffles) breaks the variant-outcome relationship by randomly shuffling labels, building a null distribution. The p-value is the proportion of permuted differences as extreme as the observed difference. The analysis also demonstrates the infer package for a cleaner tidy workflow.

Key Code

# Bootstrap CI for absolute lift
boot_diffs <- replicate(4000, {
  p_c <- mean(sample(control_rows$converted_num, replace = TRUE))
  p_t <- mean(sample(treat_rows$converted_num, replace = TRUE))
  p_t - p_c
})
boot_ci <- quantile(boot_diffs, probs = c(0.025, 0.975))

# Permutation test
perm_diffs <- replicate(4000, {
  df_perm <- df %>% mutate(variant_perm = sample(variant))
  # ... compute diff under shuffled labels
})
perm_p_value <- mean(abs(perm_diffs) >= abs(obs_diff))

Visualization

Results

Bootstrap Results

Permutation Results

Method Comparison

Key Findings

• Bootstrap CI [+0.12pp, +2.57pp] closely aligns with the analytical CI from prop.test [+0.11pp, +2.55pp] – the methods converge
• All three p-values (0.032, 0.036, 0.033) are consistent and significant at α=0.05 – strong evidence the result is not due to chance
• The bootstrap distribution provides a direct visual of lift uncertainty – more intuitive for stakeholders than a p-value alone

Business Recommendations

• Bootstrap CIs are increasingly preferred in industry – they are intuitive, assumption-free, and work well with any metric distribution
• Use the bootstrap distribution as a stakeholder visual: "here is what the lift looks like across 4,000 resamples of the data"
• Permutation tests directly answer "could this result be due to chance?" – a compelling framing for non-technical decision-makers

Approach

Regression adjustment – known as CUPED (Controlled-experiment Using Pre-Experiment Data) in industry – uses pre-experiment covariates to reduce residual variance, producing tighter confidence intervals without requiring additional sample size. This is standard practice at Microsoft, Netflix, and Uber for improving experiment sensitivity.

For the primary metric (conversion), the analysis fits a logistic regression with variant and log1p(pre_sessions_7d) as predictors, then computes an adjusted risk difference via marginal means (G-computation). HC3 robust standard errors from the sandwich package protect against heteroskedasticity. A bootstrap CI (B=1,000) provides distribution-free uncertainty for the adjusted risk difference.

For the guardrail metric (time-to-complete), an OLS regression with the same covariates is used. The treatment coefficient directly estimates the adjusted mean difference in minutes, again with HC3 robust standard errors.

Key Code

# Logistic regression for conversion
m_conv <- glm(converted ~ variant + log1p_pre_sessions_7d,
              data = df, family = binomial())
vc_conv <- sandwich::vcovHC(m_conv, type = "HC3")

# Adjusted risk difference via marginal means
p_control <- predict(m_conv, newdata = df %>% mutate(variant = "control"),
                     type = "response")
p_treat   <- predict(m_conv, newdata = df %>% mutate(variant = "treatment"),
                     type = "response")
adj_rd <- mean(p_treat - p_control)  # Adjusted lift in probability

# OLS for guardrail metric
m_time <- lm(time_to_complete ~ variant + log1p_pre_sessions_7d, data = df)
vc_time <- sandwich::vcovHC(m_time, type = "HC3")

Results

Logistic Regression Coefficients (Conversion)

Standard errors: HC3 robust via sandwich package.

Adjusted Risk Difference (Conversion)

OLS Regression Coefficients (Time-to-Complete)

Standard errors: HC3 robust via sandwich package.

Key Findings

• Adjusted conversion lift (+1.33pp) matches the unadjusted estimate closely – the covariate adjustment confirms rather than changes the conclusion
• The adjusted bootstrap CI [+0.25pp, +2.57pp] is slightly tighter than the unadjusted CI, demonstrating the variance-reduction benefit of CUPED
• The pre-experiment covariate (pre_sessions_7d) is not a significant predictor of either outcome (p > 0.38) – expected in a well-randomized simulation, but the adjustment still provides marginal precision gains
• Adjusted time-to-complete difference (−0.341 min) is consistent with the unadjusted estimate, with robust standard errors protecting against heteroskedasticity

Business Recommendations

• Regression adjustment should be standard practice – it provides free precision (tighter CIs) without requiring more users or longer experiments
• Always use robust standard errors (HC3) for experiment analysis – they protect against model misspecification and heteroskedasticity
• Report the adjusted risk difference in percentage points – stakeholders understand "1.33pp lift" better than "odds ratio of 1.148"

Approach

Power analysis answers a critical pre-experiment question: "given our sample size, what is the smallest effect we can reliably detect?" This minimum detectable effect (MDE) determines whether an experiment is worth running. If the MDE is larger than the expected effect, the experiment is underpowered and likely to produce inconclusive results.

The analysis calculates the MDE at 80% power for the current sample size, computes required sample sizes for a range of target lifts (0.5pp, 1.0pp, 1.5pp, 2.0pp), and generates a power curve showing how statistical power changes across effect sizes.

Key Code

# MDE given current sample size
mde_abs <- mde_solver(p0 = baseline_rate, n = n_per_group,
                       alpha = 0.05, power_target = 0.80)

# Required sample sizes for target lifts
power.prop.test(power = 0.80, p1 = baseline_rate,
                p2 = baseline_rate + target_lift, sig.level = 0.05)

# Power curve across effect sizes
power_curve <- tibble(
  lift_pp = seq(0, 3, by = 0.1),
  power = sapply(lift_pp / 100, function(d) {
    power.prop.test(n = n_per_group, p1 = p0,
                    p2 = p0 + d, sig.level = 0.05)$power
  })
)

Visualization

Results

MDE Summary

Sample Size Requirements

Key Findings

• With ~5,000 users per group and a 10.19% baseline, the MDE at 80% power is 1.77pp – the experiment is slightly underpowered for the observed 1.33pp lift (power ~55% at that effect size)
• The planted 1.5pp effect falls just below the MDE threshold, which explains why the observed p-value (0.032) is significant but not overwhelmingly so
• Detecting a 0.5pp lift would require ~58,700 per group (roughly 12x the current sample) – small effects demand large experiments
• The power curve shows diminishing returns: moving from 80% to 95% power roughly doubles the required sample size

Business Recommendations

• Always calculate MDE before launching an experiment – if the MDE is larger than the expected effect, the experiment is underpowered and likely to waste resources
• Use power analysis to negotiate experiment duration: "we need X weeks at current traffic to detect a Y% lift with 80% power"
• For small expected effects, use CUPED/regression adjustment to lower the effective MDE without increasing sample size

R Packages Used

Statistical Methods Applied

Programming Practices

The pipeline follows a modular architecture: each of the 7 scripts (00–06) reads from data/, writes results to tables/ and visualizations to figures/. Defensive coding is used throughout – stopifnot() for column checks, explicit factor levels, and SRM-based early termination. All results are reproducible via set.seed(123), and the code follows a consistent tidyverse style with clear variable naming.

Pipeline Summary

Starting with data generation, the pipeline verified randomization integrity via SRM (chi-square p = 0.734), measured the primary conversion lift (+1.33pp, p = 0.032), confirmed the guardrail metric showed improvement (−0.34 min, p < 0.001), validated results with distribution-free inference (bootstrap CI [+0.12pp, +2.57pp], permutation p = 0.036), refined estimates with covariate adjustment (+1.33pp adjusted, bootstrap CI [+0.25pp, +2.57pp]), and assessed statistical power (MDE = 1.77pp at 80% power with current sample size).

Key Takeaways

• Treatment increased conversion by +1.33pp (absolute) – a ~13% relative lift over the 10.19% baseline
• Guardrail metric (time-to-complete) improved by 0.34 minutes – no regressions detected
• All inference methods (prop.test, bootstrap, permutation) produced consistent results (p-values: 0.032, 0.036, 0.033)
• Regression adjustment confirmed the unadjusted estimate and tightened the confidence interval
• The experiment was slightly underpowered (MDE = 1.77pp vs. observed 1.33pp) but still detected a significant effect
• The pipeline follows industry best practices: SRM → primary metric → guardrails → validation → regression adjustment → power

Limitations

The data is simulated – real experiments involve messier realities including non-compliance, network effects, and novelty/primacy bias. The analysis focuses on a single binary primary metric; real experiments often track multiple correlated metrics requiring multiple comparison correction (e.g., Bonferroni, Benjamini-Hochberg). Future enhancements could include sequential testing for early stopping, Bayesian analysis with informative priors, and heterogeneous treatment effect estimation (CATE) to identify which user segments benefit most.