Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add a page to explain there are two types of interactions #6

@lukasvermeer

Description

@lukasvermeer

Interactions between two experiments can be of one of two kinds: traffic or metric interactions.

Traffic interactions
Traffic interactions occur when one experiment treatment is causing more or less traffic to flow to another experiment.

For example, when variation 1 of experiment 1 changes the destination of a link on the homepage from the one page to another, while experiment 2 is making changes on one of those pages. In that case, the first experiment will cause more or less traffic to flow into the second experiment.

Traffic interactions are not necessarily a problem. The traffic in experiment 2 in the example above can still be equally split and the metrics can still be tracked. There is however a risk of bias, since the traffic diverted by experiment 2 might be different from the remaining traffic in some meaningful way.

Metric interactions
Metric interactions occur when the impact on a given metric for a combination of two experiments is different from what we see in either experiment in isolation. For any two experiments, there might be differences in interaction effects for different metrics.

For example, when experiment 1 shows an uplift of 5% on conversion and experiment 2 shows an uplift of 10% on conversion, then we would expect the combination to show an uplift of approximately (1,1*1,05)-1 = 15,5%. If instead we see a drop of 50% for the audience which is exposed to both tests, then we would call that a (in this case negative) interaction. At the same time, for the same experiments, we might not see any such interaction effect on CTR or other metrics.

Similar to traffic interactions, metric interactions are not necessarily a problem depending on the nature and severity of the interaction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions