A/B Testing of Web Apps using Traffic routing feature

Introduction

A/B Testing feature gives you an amazing way to test new website content, processes, workflows, etc. by routing the production traffic into multiple slots. At a very high level, you split your website’s traffic between two deployments of code and measure the success of each version of the site based on your needs. Normally it requires some 3rd party tools to be deployed to achieve this simple looking (but actually much more complex) feature. However, Azure App Service enables us to set up it very quickly with the help of “Testing in Production Feature” & “Deployment Slots”.

Setup & configuration

To start with it, we need at least 2 deployment slots (A & B) for the Web App. I’ve created a “To-Do list creation “Java app following this tutorial and deployed it on the default (production) slot. To clearly differentiate, I added “Production” Label on my home page as below

Production slot URL: https://todoappjava.azurewebsites.net/

As our Production Slot is ready, we would need slot “B” for A/B testing. Typically in real world scenario, this would be the slot having new features/enhancements for which you want to judge end user’s experience.

As we already explained in the blog here, I created a slot called “staging” and deployed updated build of my App by having some minor change like, addition of creation date column in task list & making the site label to have “Staging” text to quickly recognize the difference.

My slot setting looks like below:

B Slot URL: https://todoappjava-teststaging.azurewebsites.net/

Setup & configuration

Now we have our 2 slots (production & staging) ready for A/B testing. To configure the traffic routing rule, navigate to Web App blade of your production slot > Testing in Production. This will display existing traffic routing set up for the slots. As we can see, by default all the traffic is getting routed to production slot.

You can change the traffic % as you wish and save the configuration. Your site is configured for A/B testing. Production Traffic would be adjusted & users can now experience the new features based on the rule settings.

Testing

Testing of this feature can be done in a number of ways from automated tools to simply hitting refresh in the browser. For this tutorial I created a quick and simple console application that will hit the production site, count the number of times each version comes up, and calculates the percentage for me. The code is very simple and uses HttpClient to hit production slot URL and check the response content & prints % hits for prod and staging slots on the console.

Running this app displays all the information I need to ensure the Traffic Routing is properly configured and the percentages are in line with my setup.

############################# Result ##################

No of Hits : 50, Prod hit 25 times, Staging hit 25 times

############################# End #####################

In actual cases, testing methods would be much deeper and can leverage App Service feature such as “Application Insights” to capture matrices on User behavior.

Conclusion

In this post, we saw how you can very quickly and simply set up A/B testing on any website hosted on Azure App Services. A/B testing provides site owners an amazing way to test new features, layouts, and more. In the next blog we will explore the setup and usage of “Application Insights” extension in Web App.