A/B Testing and Testing In Production with Azure Web Apps

I've got a lot of production web sites running in Azure right now. Some are small for side projects and some are larger like the sites for the Hanselminutes Podcast and This Developer's Life. I like Web Apps/Sites (which is Platform as a Service) rather than Virtual Machines (Infrastructure as a Service) because I don't like thinking about the underlying operating system if I can avoid it. I like to be able to scale the site up (faster, bigger) or out (more machines in the farm) with a slider bar.

There's some other more advanced and useful features that Azure Web Apps have that keep me using them.

I'll use a little site I made called KeysLeft.com that tells you how many keystrokes are left in your hands before you die. Think of it as a productivity awareness tool.

First, I'll add a Deployment Slot to my existing Git-deployed Web App. The source for KeysLeft lives in GitHub here. When I check-in a change it's automatically deployed. But what if I wanted to have a staging branch and automatically deploy to a staging.keysleft.com first? If it works out, then move it to production by swapping sites. That'd be sweet.

Staging Slots for Azure Web Apps

You can see here my main KeysLeft web app has a Staging "side car" app that is totally separate but logically related/adjacent to production.

This Web App has its configuration copied from the main one, and I can setup Continuous Deployment to pull from a different branch, like "staging" for example. The name of the deployment slot becomes a suffix, so keysleft-staging.azurewebsites.net unless you set up a custom CNAME like staging.keysleft.com. You can have up to 4 deployment slots in addition to production (so dev, test, staging, whatever, production) on Standard Web Apps.

A/B Testing for Azure Web Apps

Once I've got a slot or two setup and running a version of my app, I can do A/B testing if I'd like. I can setup a feature that was called "Testing in Production" and is now "Traffic Routing" and tell Azure what percentage of traffic goes to prod and what goes to staging. Of course, you have to be sure to write your application so such that authentication and session is managed however is appropriate, especially if you'd like the user to have a seamless experience.

Here I've got 10% of the traffic going to staging, seamlessly, and the other 90% is going to production. I can make a small change (background color for example) and then hit the main site over and over and see the occasional (10% of course) request being routed to the staging slot. You can configure this static routing however you'd like.

Then I could hook up Application Insights or New Relic or some other event/diagnostics system and measure the difference in user reaction between features that changed.

Advanced Testing in Production

Made it this far? Then you're in for a treat. Static routing is cool, to be clear, but scripting a more dynamic experience is even more interesting. Galin Iliev, one of the developers of this feature, gave me this Powershell script to show off more powerful stuff.

First, you can use PowerShell to manage this stuff. You can change routing values and ramp up or ramp down. For example, here we start at 10% and change it by 5 every 10 minutes.

# Select-AzureSubscription YOURSGOESHERE

$siteName = "keysleft"
$rule1 = New-Object Microsoft.WindowsAzure.Commands.Utilities.Websites.Services.WebEntities.RampUpRule
$rule1.ActionHostName = "keysleft-staging.azurewebsites.net"
$rule1.ReroutePercentage = 10;
$rule1.Name = "staging"

$rule1.ChangeIntervalInMinutes = 10;
$rule1.ChangeStep = 5;
$rule1.MinReroutePercentage = 1;
$rule1.MaxReroutePercentage = 80;

Set-AzureWebsite $siteName -Slot Production -RoutingRules $rule1

But! What if you could write code to actually make the decision to continue or fall back dynamically? You can add a callback URL and a Site Extension called the "TiP Callback Extension."

$rule1.ChangeDecisionCallbackUrl = "https://keysleft.scm.azurewebsites.net/TipCallback/api/routing"

This callback gets a small POST when it's time to make a decision. The Site Extension (and all Site Extensions for that matter) is just a little sidecar Web API. You can check out the code by visiting yoursite.scm.azurewebsites.net and look at the Site Extensions folder. Here is the idea. You decide what to do based on HTTP-related context that was passed in and then return a ChangeDirectionResult object.

using System.Web.Http;
using TipCallback.Models;

namespace TipCallback.Controllers
{
    public class RoutingController : ApiController
    {
        [HttpPost]
        public ChangeDirectionResult GetRoutingDirection([FromBody] RerouteChangeRequest metrics)
        {
            // Use either Step or RoutingPercentage. If both returned RoutingPercentage takes precedence
            return new ChangeDirectionResult
            {
                Step = (int)metrics.Metrics["self"].Requests,
                RoutingPercentage = 10
            };
        } 
    }
}

Here's the object you return. It's just a class with two ints, but this is super-annotated.

/// 

/// Return information how to change TiP ramp up percentage.
/// Use either Step or RoutingPercentage. If both returned RoutingPercentage takes precedence
/// Either way MinRoutingPercentage and MaxRoutingPercentage set in API rule are in force
/// 

[DataContract]
public class ChangeDirectionResult
{
    /// 

    /// Step to change the Routing percentage. Positive number will increase it routing.
    /// Negative will decrease it.
    /// 

    [DataMember(Name = "step")]
    public int? Step { get; set; }

    /// 

    /// Hard routing percentage to set regardless of step.
    /// 

    [DataMember(Name = "routingPercentage")]
    public int? RoutingPercentage { get; set; }
}

All this stuff is included in Standard Azure Web Apps so if you're using Standard apps (I have 19 websites running in my one Standard plan) then you already have this feature and it's included in the price. Pretty cool.

Related Links

• Video: Intro to Testing in Production
• Video: Scripting Testing in Production and Traffic Routing
• Video: Azure WebSites - Deployment Slots for Staging Sites
• All the Azure Friday Web Sites/Web Apps videos!

Sponsor: Big thanks to Infragistics for sponsoring the feed this week. Responsive web design on any browser, any platform and any device with Infragistics jQuery/HTML5 Controls.  Get super-charged performance with the world’s fastest HTML5 Grid - Download for free now!