A/B Testing Analysis: Extracting Insights and Drawing Conclusions

Project Overview

The A/B Testing Project aims to explore and analyze the effectiveness of a new design variant compared to an existing one through rigorous statistical analysis and experimentation. By leveraging user interaction data, the project seeks to uncover actionable insights into various metrics such as completion rates, time spent on different steps, error rates, and step abandonment rates. Through a structured approach encompassing data preparation, exploration, analysis, and statistical testing, the project aims to determine whether the proposed design changes lead to meaningful improvements in user engagement and overall user experience. With the aid of Python libraries such as pandas, numpy, matplotlib, seaborn, scipy, and statsmodels, the project navigates through the stages of data preprocessing, visualization, hypothesis testing, and experiment evaluation, ultimately drawing conclusions about the efficacy of the new design variant and providing valuable insights for decision-making.

Datasets

Datasets can be found here: https://github.com/data-bootcamp-v4/lessons/tree/main/5_6_eda_inf_stats_tableau/project/files_for_project

Project Components

1. Data Preparation and Exploration

• Merging and preprocessing datasets containing user interaction data.
• Exploring data distributions, missing values, and general trends.

2. Completion Rates Analysis

• Calculating completion rates for each variation group (control and test).
• Analyzing completion rates by gender and age groups within each variation group.
• Visualizing completion rates for better insights.

3. Time Spent Analysis

• Calculating and comparing the average time spent on each step between variation groups.
• Analyzing time spent on each step transition.
• Visualizing time spent on each step to identify efficiency improvements.

4. Error Rates and Step Abandonment Analysis

• Calculating error rates and step abandonment rates for each variation group.
• Visualizing error rates and step abandonment rates to assess the impact of the new design on user experience.

5. Statistical Testing

• Conducting hypothesis testing to determine the statistical significance of observed differences.
• Testing for completion rate increase exceeding a predefined threshold.
• Assessing group distribution equality between variation groups.

6. Conclusions and Experiment Evaluation

• Evaluating the experiment's structure, randomization, and duration.
• Providing insights into the reliability and validity of the experiment results.

Project Structure

• Analysis.ipynb: Jupyter Notebook containing the Python code for data preparation, analysis, visualization, hypothesis testing, and experiment evaluation.

Python Libraries Used

• pandas
• numpy
• matplotlib
• seaborn
• scipy
• statsmodels

Presentation

Link to Presentation: https://1drv.ms/p/s!Ao0pVqWLGOAwl2XkqSWOgRAEEeGf