Experimentation in UpGrade


UpGrade is an open-source software for A/B testing in educational software. This guide shows how you can host UpGrade for your organization, and integrate it with your educational platform.

How you can use UpGrade for experimentaiton?

say you have an edtech application and you want to improve it scientifically using UpGrade. How do you go about doing this?
Run UpGrade somewhere (locally or on the cloud)
Add 'decision points' in your app (read more in key concepts page)
Connect your app with UpGrade via client libs
Run experiments on your decision points in the code using UpGrade

A walkthrough of the UpGrade Web Application

The web portal of UpGrade provides an easy way to manage experiments in the educational software. Let us look at the primary feature of experiment creation in UpGrade.
(Please follow the guide in the later half of this article to know how to host UpGrade on your cloud infrastructure.)
After logging in to UpGrade using your credentials, you can start creating an experiment.
On the home page, you get two options to create an experiment. You can either import an experiment by clicking on the Import Experiment button and uploading the experiment design JSON file that you might have exported earlier or click on the Add Experiment button to create a new experiment.
There are two options provided to export experiment details. You can either export the experiment design in JSON format by clicking on EXPORT DESIGN button or export the experiment data in CSV format by clicking on EXPORT DATA button.
If you want to add a new experiment, in the first step, you have to enter the name of the experiment and some description (optional). After that, you have to set the Unit of Assignment for the experiment. UpGrade allows users to be randomly assigned to different conditions both at the individual and group level. In the group assignment, you can ensure that all of the users within the group get the same condition assignment. If you choose to assign the conditions at the group level, you have to select the group type. Custom group types are also allowed. Else you can also choose to assign the conditions at the Individual level. The Consistency Rule of the experiment is used to ensure that either everyone in the group gets the same assignment or they all remain excluded from the experiment. You can also attach contexts and tags to the experiment. The context can be used on the application side to only fetch relevant experiments.
We are setting up an experiment that compares the effectiveness of two different versions of the same lesson. To design this experiment, you have to create two conditions in the Experiment Design tab, and give each of them an assignment weight. The assignment weights should add up to 100. After the conditions, you have to define Experiment Sites for the experiment. Each site is represented by its corresponding Experiment Point and ID. Experiment Point is the place or function in code where the conditional code execution happens. For more description, please refer to the following papers: UpGrade: An Open Source Tool to Support A/B Testing in Educational Software and Optimizing an Educational Game Using UpGrade: Challenges and Opportunities.
You can choose to start the experiment at the time of its creation or schedule it to start any time in the future. You can end the experiment based on a condition too.
At the end, you can set the rules for what will happen when the experiment will end. You can either continue to give users the same condition they were in during the experiment, or change their final conditions to a default non-experiment condition.

A Diagram showing a typical integration scenario