In this analysis I determine if changing the color of the "buy" button for an online retailer had an impact on sales
An online retailer is curious if there would be a bump in sales by changing the color of the "buy" button on thier site from red to yellow. The sales department believes that the red "buy" button blends in too much with the web page and deters some consumers from completing a purchase. The retailer has decied to perform an A/B test to determine if a shift from a red "buy" button to a yellow "buy" button will show an increase in the purchase rate. In this analysis I will perform the exploratory data analysis on 9,000 visits to the retailer as a baseline then conclude from the A/B test of 1,000 randomized visits to the site if the color change made an impact on the number of purchases.
(Spoiler Alert!: The results of the A/B test failed to provide statistically significant evidence that change in color from red to yellow impacted the sales rate.)
The baseline data (pre A/B test) consisted of 9,000 visits to the site that result in a purchase or not. In the pre-test data there were 4,730 (53%) visits that resulted in a purchase and 4,270 (47%) that did not. With each observation the following data was collected:
time_on_page_sec: The ammount of time in seconds an individual user spent on that page
num_product_reviews: The number of product reviews for the product the customer viewed
product_rating: The product rating of the product the customer viewed
purchase: An indicator of whether or not the customer purchased the product
In the A/B test 1,000 visits to the site were randomly assigned to either an experimental group who saw a yellow "buy" button or the control group who saw a red "buy" button. The experimental group contained 504 visits and the control group contained 496 visits for an almost completely balanced experiment. The experimental group resulted in 172 (34%) purchases, while the control group resulted in 138 (28%) purchases. The same independent data was collected during the A/B test as in the pre A/B test.
The first thing to note is that in the pre A/B test the number of purchases seems about evenly split.
When we take a closer look at the number of purchases and the independent variables it becomes apparent that there is a difference in the number of product reviews and product rating for the visits that resulted in a purchase and those that did not. We can infer from this that the amount of time spent on the page does not indicate if the visit will result in a purchase. We can also infer that the actual product being viewed has a big impact in determining if the visit will result in a purchase. The more reviews a product has and a higher rating lead to a larger probablity of a purchase occuring.
Hypothesis tests confirmed that there is a statistically significant difference in the average number of product reviews and product rating for visits that resulted in a purchase and those that did not. Tests also confirmed there is no difference in the average amount of time spent on the page. All tests were conducted with an α = 0.05.
The scatter plots using time spent on page as the x-axis and number of reviews and product rating as the y-axis shows there are no signs of a high correlation with the exception of the negative correlation between time and product rating. We can infer from this that the higher a product is rated, the less time it takes for a customer to decide to purchase.
The analysis thus far has shown that a purchase is heavily influenced by the number of reviews and rating of the product.
When categorizing the number of reviews into groups, we see less separation between the visits that resulted in a purchase and those that did not. It is only in the middle category where products have 31 to 60 reviews that there is noticeable difference.
When categorizing the product ratings, we see the same trend as we did with the number of reveiws. There appears to only be a separation between the visits that resulted in a purchase or not in the middle category.
The first thing to note when conducting this analysis is that we are working with a binomial distribution. A visit can have 1 of 2 outcomes, purchase or not, with an assumed fixed probability of a purchase. We will use this distribution to perform the hypothesis test needed to provide a decisive answer to the retailer. From this test we want to know if the purchase rate from the visits in the experimental yellow "buy" button group was higher than those in the control group with the original red "buy" button. Statistically speaking we will test the following null hypothesis against the alternative hypothesis:
To do this we will first need a baseline probability of purchase from our Pre A/B test data. To do this, I've taken 30 random samples of 500 data points with replacement of our pre A/B test data and recorded the purchase rate from each. 500 data points were used in each subset as that is the approximate size of our experimental and control group. The scatter plot below shows the purchase rate from these samples. An average purchase rate of approximately 0.52. This will provide our probability of a purchase and help determine the variance needed to conduct our hypothesis tests.
For our hypothesis test we will assume that the experimental and control groups had equal variances that are known from our pre A/B test data. Using the binomial distribution our variance takes the form of np(1-p). From our resampling of the pre A/B test data we have p = 0.52 (probability of purchase) and (1-p) = 0.48 (probability of no purchase). From our experimental group we had 504 visits to the site with 172 that resulted in a purchase. In the control group there were 496 visits and 138 resulted in a purchase. Thus to get our test statistic we will have the following:
The test statistic of 0.09 gives a p-value of 1 - 0.5359 = 0.4641 which does not provide evidence to reject the null. Hence we do NOT have evidence to reject the null hypothesis & can conclude that there is not a statistically significant difference in the purchase rate between the experimental and control groups.
To control the effects of the product under consideration during the visit, the same method was conducted with the same number of product review categories ("30 or Fewer Reviews", "31 to 60 Reviews", and "61 or More Reviews") and product rating categories ("Product Rating <= 2", "Product Rating Between 2 & 3", "Product Rating > 3"). These tests resulted the same as our overall test. There were no significant differences found in the purchase rates between the experimental and control groups. These tests have been omitted for brevity's sake, but can be found in the R-code within this repository.
The results of the A/B test did not show statistically significant evidence to support the sales department's hunch that a yellow "buy" button would lead to more sales than the current red "buy" button. It is unlikely that making this change would lead to any additional sales, but it is possible that a color change could lead to a better customer experience. From the analysis we saw that higher rated products and products with more reviews did have an impact on the number of purchases. The online retailer can take the learnings from this test and commit to another test on different methods to incentivize its customers to leave more reviews. The retailer could also test in the future if different suppliers can provide products for their site that lead to higher ratings. Increasing the number of product reviews and higher rated products should be the objectives of future A/B tests to improve sales.