pyAB API

(Version 0.0.1)

Experiments

class pyab.experiments.ABTestBayesian(success_prior, trials_prior)

Bayesian A/B Testing.

Parameters:
  • success_prior (int) – Number of successful samples from prior.
  • trials_prior (int) – Number of trials from prior.
calculate_uplift_area()

Calculate Uplift pdf & area beyond threshold.

Returns:
  • uplift_distribution (ndarray) – uplift distribution based on chosen uplift method.
  • uplift_area (float) – percentage area above threshold.
conduct_experiment(success_null, trials_null, success_alt, trials_alt, uplift_method='uplift_percent', num_simulations=1000)

Conduct experiment & generate uplift distributions.

Parameters:
  • success_null (int) – Number of successful samples for variant-a.
  • trials_null (int) – Number of trials for variant-a.
  • success_alt (int) – Number of successful samples for variant-b.
  • trials_alt (int) – Number of trials for variant-b.
  • num_simulations (int) – Number of mcmc simulations.
  • uplift_method (str, default = 'uplift_percent') –

    Uplift evaluation metric.

    • ’uplift_percent’:
      percent uplift gain from variant-a to variant-b
    • ’uplift_ratio’:
      uplift ratio of variant-b & variant-a
    • ’uplift_difference’:
      uplift difference between variant-b & variant-a
plot_uplift_distributions(figsize=(18, 6))

Plot uplift pdf & cdf for provided experiment parameters.

Parameters:figsize (tuple, default = (18, 6)) – matplotlib plot size.
print_bayesian_results()

Print Bayesian Experiment Results

class pyab.experiments.ABTestFrequentist(alpha=0.05, alt_hypothesis='one_tailed')

Frequentist A/B Testing aka Two sample proportion test.

Parameters:
  • alpha (float, default = 0.05) – Significane level or Type 1 error rate.
  • alt_hypothesis (str, default = 'one_tailed') –

    One or two tailed hypothesis test.

    • ’one_tailed’:
      one tailed hypothesis test
    • ’two_tailed’:
      two tailed hypothesis test
calculate_power(stat)

Calculate power (1-beta) at given test statistics.

Parameters:stat (float) – z or t test statistic.
Returns:1 - beta – power at given test statistic.
Return type:float
calculate_stat(prop_alt)

Calculate test statistic with current experiment parameters.

Parameters:prop_alt (float) – alternate hypothesis proportion.
Returns:stat – z or t statistic.
Return type:float
conduct_experiment(success_null, trials_null, success_alt, trials_alt)

Conduct experiment & generate power curve with provided parameters.

Parameters:
  • success_null (int) – number of successful clicks or successful events (Version-A).
  • trials_null (int) – number of impressions or events (Version-A).
  • success_alt (int) – number of successful clicks or successful events (Version-B).
  • trials_alt (int) – number of impressions or events (Version-B).
Returns:

  • stat (float) – z or t statistic.
  • pvalue (float) – probability of obtaining results atleast as extreme as the results actually observed during the test.
get_sample_size(beta=0.1)

Calculate required sample size per group to obtain provided beta.

Parameters:beta (float) – Type 2 error rate.
Returns:n – sample size per group.
Return type:int
plot_power_curve(figsize=(9, 6))

Plot power curve for provided experiment parameters.

Parameters:figsize (tuple, default = (9,6)) – matplotlib plot size.
print_freq_results()

Print Frequentist Experiment Results