An AB test sample size calculator. Calculations come through basic SciPy.stats methods, the web app is built with Streamlit and hosted on Heroku.
See it live here.
See also my AB test significance calculator.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
Clone/fork the repo onto your local machine.
It is then recommended to use a virtual environment to install the dependencies using the requirements.txt file.
pip install -r requirements.txt
With these installed, you simply need to run
streamlit run app.py
Alternatively, with Docker, use the following command and then navigate to localhost.
docker run -dp 80:8080 ryanfox212/samplesize
I utilised Heroku's free tier to host the web app. To do the same, create a project on Heroku's website and then I recommend linking the project to a Github repository in the Deploy section.
Alternatively, you can use the Heroku CLI. Provided you have already created a project on the site, use
heroku git:remote -a [project-name]
git push heroku master
Use the same project name as the one you created. If successful, you can then go to https://[project-name].herokuapp.com/.
- Streamlit - The web application framework used
- SciPy - For the statistical methods
- Seaborn - For vizualisations
- Heroku - Cloud platform used for deployment
The calculator asks for daily observations and conversions and the number of variants that you would like to test.
Optional inputs:
- business value
- significance level (1-alpha)
- statistical power
- maximum runtime
A notable exception from the possible inputs is minimum detectable effect that you hope to achieve through the change made. Instead, this sample size calculator outputs a range of MDEs based on how long you run the test.
Personally, I have found this form of output more useful when scoping out tests to understand the level of precision that might be achieved.
The number of visitors I need in my test to achieve the required sample by itself does not tell me very much. It is this number relative to my traffic levels that gives me the information I need.
Adding the business value of a conversion helps to understand what is the potential benefit/risk behind the change.
This project is licensed under the MIT License - see the LICENSE file for details.