Skip to content
Fitness News You Can Trust

A Different Approach to Statistical Analysis in A/B Testing

Performing an A/B test entails determining if two groups of observations stem from distinct distributions (say P and Q). The sequential steps include establishing a null and an alternative hypothesis (Null: P=Q, Alternative: P≠Q); determining a significance level alpha; and constructing a...

,   and  Administrator
3 min read
A Different Approach to Statistical Analysis in A/B Tests
A Different Approach to Statistical Analysis in A/B Tests

A Different Approach to Statistical Analysis in A/B Testing

In the realm of A/B testing, where the goal is to compare two distributions, such as the behavior of group A versus group B in an experiment, the use of High Probability Lower Bounds (HPLBs) has emerged as a valuable tool. These bounds provide statistically valid, lower confidence limits on quantities of interest, such as the total variation distance between two distributions, with guaranteed high confidence.

The Role of HPLBs

HPLBs serve a crucial function by offering a provable lower bound on the total variation distance between two groups with a specified high probability. The total variation distance quantifies the maximum difference across all events between the probability distributions of the two groups. In other words, an HPLB guarantees that the true difference is at least as large as the bound, with a controlled risk of error.

Implications of HPLBs

Sequential Testing and Optional Stopping

Unlike classical fixed-horizon tests, HPLBs (via corresponding e-processes) allow experimenters to stop early or analyze results as data arrives without inflating Type I error rates (false positives). This is particularly important for A/B tests run over time where continuous monitoring is practiced.

Robustness and Validity

The e-processes underlying HPLBs are constructed such that for any stopping time, the probability they overstate the difference beyond a certain threshold is bounded by a chosen significance level (\alpha). This property is stronger than standard confidence intervals, which often do not hold under arbitrary stopping.

Application to Total Variation Distance

Since total variation distance is a key metric to quantify how two distributions differ, having high-probability lower bounds on it means an experimenter can confidently claim at any point in the test that the difference between the variants is at least a certain amount. This can guide decisions about whether a variant's effect is meaningfully different from control.

In essence, HPLBs serve as a safe, anytime-valid statistical tool that provides guaranteed lower confidence bounds on the magnitude of distributional differences in A/B tests, particularly the total variation distance. This allows for reliable and flexible decision-making without inflating false positives due to repeated data peeking or early stopping.

Key Features of HPLBs

  • The value 0.64, for example, will be a lower bound for the true total variation distance with high probability.
  • The confidence level (1-α) for the HPLB calculation can be defined.
  • The total variation distance is the right notion for the HPLB in A/B testing.
  • The general steps of an A/B test include deciding between two hypotheses, defining a level of significance, constructing a statistical test, deriving a test statistic, and obtaining a p-value.

Advantages of HPLBs

  • The HPLB can detect small differences between the distributions, even when the difference is as small as 0.05.
  • Each value larger than zero means a test that P=Q got rejected on the 5% level.
  • The estimate λˆ, derived from the HPLB, entails both the statistical significance and the effect size estimation in A/B testing.
  • The HPLB provides an integrated answer to the difference in distribution and the intensity of the shift.

Detection Capabilities of HPLBs

  • The HPLB can detect when there is no change in the two distributions (when the difference is zero).
  • The HPLB can detect that the difference between the distributions is larger the larger the difference is.
  • This lower bound is named the high probability lower bound (HPLB).

For more details and comparisons to existing tests, please refer to the paper and the R-package HPLB on CRAN. Loris Michel and Jeffrey Naef are contributors to this article.

  1. High Probability Lower Bounds (HPLBs) not only offer a provable lower bound on the total variation distance between two groups with a specified high probability, but they also serve as a safe, anytime-valid statistical tool that provides guaranteed lower confidence bounds on the magnitude of distributional differences in A/B tests, particularly the total variation distance.
  2. In the field of health-and-wellness, science, and therapies-and-treatments, HPLBs can be utilized to ensure that decisions about the effect of a variant in an A/B test are reliable and based on a guaranteed, statistically significant difference, without the risk of inflated false positives due to repeated data peeking or early stopping.

Read also:

    Related

    Latest