A/B Tests versus Canary releases
Data analysis -a field with measurement as a core pillar- can only reveal trends and compute statistics on gathered data. When there is no representative data of the future to explore and extract insights from,how do we make decisions? How do we know what the future holds? Instead of predicting, why don’t we test? Better still, why don’t we predict and test?
Sample scenarios are :
- gauging the response from expansion into a different region.
- assessing inclusive strategy for a new customer segment
- testing a machine learning model on live data.
By simulating or replicating states and conditions to our possible best, we can attempt to preview impact.
This experimental approach answers questions. For example I have a million dollars to spend on Ads. How do I maximize my spending; Will changing this copy improve my conversion rate? Is the variant of this drug effective before we begin to mass produce? This new iteration of our model/product is ready. What does the market think?
A/B testing
Meaning can be inferred from the name(A or B), but implementation and intricacies go beyond the surface level.
In A/B speak, A is a control group while B is the treatment. Both groups curated by sampling. The variable to be tested(example) is the only difference between A and B. After the duration of the experiment, results are compared, hypothesis made, and conclusion inferred. When more than one variable is involved, incremental changes are made, variable by variable. Testing multiple variables at once can lead to errors in interpretation due to the interaction between effects of each variable. We cannot authoratively conclude that change in x led to y.
X and Y, broadly referring to input and impact, can be in terms of ads run and new website visitors, redesign and sales conversion, government policy and citizen’s acceptance, manufacturer’s design and nature of reviews/complaints, even biologically how diet/sleep routines/profession impacts a person’s effectiveness. Information provided by such tests can be immense when amplificative factors such as scale, word of mouth, health exist.
Diving deeper, when conducting A/B tests, there are guard rails to reduce the error of wrong conclusions. These guard rails are the sampling methods, chosen significance level, use of appropriate statistical tests and techniques. In Addition, limiting and disruptive factors exist in application to real-world scenarios. We will explore this later in the article.
Canary releases
In a similar light, there is an approach common in the product management/Mlops lifecycle termed canary releases(adopted from coal miners who used canaries to foretell the level of toxic elements in the air while mining). In practice, this means that products/models are gradually released to a subset of users to gather information on product performance. Information may include Performance on subpopulations/user groups, unintended consequences etc.
At the risk of sounding inhumane, the first set of users is the canaries, a form of experimentation where analysis of errors and implementation of lessons learned happens before full rollout or reversal.
Similarities
What is common to these two approaches is the need for sampling. Sampling is a vital part of A/B testing & Canary releases. If a sample treatment group is not representative of the population in the sense that attributes of both groups differ, it reduces confidence in the results.
To prevent this,you need to follow standard practices when sampling. Methods of Sampling include : random, systematic sampling, stratified, weighted or cluster. Convenience sampling however should be avoided.
Differences
Canary releases assist in assessing product versions for progressive rollout. A variant of canary release, usually applied in machine learning, is called shadow release. Two models(old and new) are present in production, but the prediction of one is only displayed, while the other is for error analysis and improvement. The reason is that integration tests are the ultimate tests of a machine learning model’s capability rather than validation tests due to possible overfitting from repeated evaluations or data leakage.
A/B testing previews the impact of decisions/changes on a particular target and is more suited to Data science/analysis tasks.
More examples -Academics may want to assess what learning formats students prefer for complex concepts. Biomedical researchers also use A/B tests in clinical trials.
The applications are numerous — evidence of its effectiveness when orchestrated correctly. Of course, not every decision can be A/B tested, but where dilemmas exist, A/B tests can be a final check.
Evaluation
A/B tests check if there’s a sizeable effect between groups using statistical tests. These tests provide p-values/ used to conclude a hypothesis given a significance level. P-values coupled with the chosen significance level and type of test (one-tailed or two-tailed)
determine the validity of the null hypothesis, Ho, of an experiment.
For instance, Correlations between numerical variables using Pearson/spearman correlation tests. It is worth pointing out that the type of statistical tests/techniques employed depends on the metric of interest.
Other statistical tests are :
- Independent or paired t-tests to compare averages of two groups — Independent t-tests where there is no relationship between the two groups. For example, assessing differences by sex(Male or female). Paired t-tests when there is a relationship between groups. An instance is assessing differences by age group, time (before and after), motivation levels at entry level and mid-senior level
- Chi-square tests to compare relationships categorical variables.
- ANOVA tests are often employed in the comparison of multiple groups
- Regression analysis and scatterplots are also used to analyse the relationship between numerical variables.
-Normality tests to examine the skewness of a sample/population distribution. This test is vital because certain assumptions apply to certain distributions.
It is worth confirming early before cascading errors happen.
As discussed earlier, considerations in sampling methods, sample size and significance level impact the effectiveness and validity of an experiment. Increased sample size increases the power statistics and reduces the margin of error. Similarly, higher significance levels correspond to a lower chance of Type II errors but lead to a higher probability for Type I errors. Therefore, these factors, in addition to the concept of P — hacking should be noted while selecting a significance level (the standard is 0.05 for one-tailed tests and 0.025 for two-tailed tests),
Canary releases
Canary release or shadow mode deployment employs a standard predefined metric guided by the purpose of the experiments for evaluation. The choice of metric may be related to business goals, user outcomes or model properties, but consistency is paramount across versions for ease of comparison.
For machine learning, this method provides the opportunity for error analysis post-deployment. The added benefit is more insight, new features and/or labels, and critical analysis of errors that a confusion matrix may not provide. The outcome is gradual deployment with improving confidence and stability with a resolution of bugs, or worst case , reversal of versions.
Canary release prioritises deployment with little to no disruption in user experience. Think of road maintenance companies and traffic management.
Limitations
It is relatively easier to perform A/B testing in theory than in practice, especially in sciences where creating treatment and control groups may not be as straightforward as in software products due to the ease virtualization affords tech-centric products.
Let’s assume the problem of obtaining treatment and control groups is solved. Other headaches exist with A/B testing. Psychological effects are unaccounted for (Novel effects — people drawn to new things, Primacy effects — people feel safer and inclined to old ways, repeated experimentation can lead to the law of diminishing marginal returns), external conditions unaccounted for may disrupt equivalence of treatment and control groups (weekends may mean people have more time for their phones, thus clicking on more ads).
Complete isolation of groups may sometimes be impossible due to unexpected interaction between groups. For example, Beta tested Ad served to a userA in a treatment group who then shares elements of that ad through social media or other means with user B in the control group(who is not supposed to see the ad). userA’s action in this way skews the final result.
This example depicts how A/B tests, designed to test specific effects, can be altered by an interaction between groups. Be sure to examine other possible effects meticulously concerning your use case.
As with A/B tests, there are also limitations in Canary releases. For one, the added computational resource required to manage multiple models in production. Additionally, Extra complexity and time for client-side ML model deployments because client permission is required. Data drift (presence of different user groups and resulting features) may lead to misleading results (possibly requiring an ensemble of models to treat each group ). Practical examples :
- multi-label classification of news documents,
- weather forecasts of different locations,
- a car’s autonomous driving model suitable for different weather conditions/terrains.
Footnote:
Power : is the likelihood of a significance test detecting an effect when there actually is one
Ho = null hypothesis, there is no effect between variables
- stratified (by defined features and categories) or weighted sampling which attributes weights to certain groups, proportional sampling means we have to collect data on every subgroup when there are lots of subpopulations with a long-tail distribution, cluster sampling
H1 = alternative hypothesis, there is effect between variables.
Statistical significance ɑ : level at which Ho is rejected if p_value is less than ɑ
Type I errors — when null hypothesis Ho is falsely rejected. This is usually When p_value is less than statistical significance.
Type II errors — when null hypothesis is falsely accepted. Error rate B which exists on the tail left side of a normal distribution.
P-hacking: P-hacking is the practice of manipulating statistical analyses to produce results that appear to be statistically significant, but are actually due to chance.
Resources for extra reading on statistics
https://www.scribbr.co.uk/stats/