Driving Unit Tests from Cloud Load Test

Visual Studio Load Testing has always made it easy to create Load Tests. Now with Visual Studio Online it is also easy to verify the performance capacity of your software of 1,000s of users external to your data centers using the Cloud and Microsoft Azure. It has been noted that Visual Studio Load Testing is widely used inside of Microsoft from its inception back in 2005; which begs the question since many of Microsoft development teams are NOT working on web pages, how are they using Visual Studio Load Test?  Many (most?) Microsoft developers are writing services and since Visual Studio Load Test can also drive Unit Tests, this gives our test teams insight into their services performance, capacity and availability.

For instance a large consumer of Visual Studio Cloud Load Test is Microsoft Games Studios and much of their code is a set of highly performant services that our game publishers consume (Billing, Scores, Profiles, etc.) For them, Visual Studio Load Test makes it easy to group unit tests into scenarios and ensure there services are performant and available, regardless the load. This Blog post will walk you through how to easily set up and test your unit tests using Cloud Load Testing. If prefer video, checkout the Load Testing video on Channel 9 where I show off load testing capabilities and also how you can use Application Insights to monitor performance.

Creating the Load Test project

Visual Load Test can be run against any Visual Studio Web Test, Coded UI* or Unit Test. Since the growing trend is for development teams to create REST based services that can be accessed from rich clients, web based clients, or even mobile devices let’s start with a sample that follows this new paradigm- specifically the web API sample: Unit Testing with ASP.NET Web API 2 which of course comes with a set of unit tests we can load test. At this point we could add a load test to the StoreApp.Tests project; but in most cases you will find reasons to not do this: such as the Unit Test project is owned by the development team and the Load Test Project owned by the test team or the fact you can NOT run your load tests with Visual Studio Online (i.e. the Cloud) if the load test is checked into an on-premises Team Foundation Server.

To add a “Web Performance and Load Test” project simply choose the File>Add New Project menu and select the Test project category.

Add the Unit Tests to the Load Test for each scenario you want to load test

Adding the Load test to your project (Right click the project > New Item > Load Test) will automatically start the load test wizard where you can add the unit tests you want to load test. While you can use data binding to test different scenarios with a single load test for readability sake most Microsoft teams create a new load test for every scenario they want to load test. In the example shown below this load test would be testing the “Get Product” scenarios.

Run your Load Test from Visual Studio Online

At this point you are ready to run your load test from your local machine. If you wanted the Load Test to simulate 10,000 users as this point you would need to acquire computers, install the needed software to run your application + the Microsoft Test Agent. This is now much easier with Cloud Load Test and to enable this all you need to do is open the Local.Testsettings file and select the option: Run Tests using Visual Studio Online. Check out MSDN for a great walk through for getting started load testing.

Collect Performance Counters from Application Insights

A challenge of most cloud-based load test solutions is the lack of access to the machine running the application under test for collecting information like performance counters. In the screen shots below you can see how to select the performance counters collected and this performance data being collected in real time. NEW With Visual Studio Update 3, Visual Studio Load Testing integrates Application Insights so collecting production data –including custom performance counters is now a piece of cake. To see more information on what sort of data and integrating Application Insights please see the MSDN documentation at http://aka.ms/aigetstarted, the Application Insights Blog at: http://blogs.msdn.com/visualstudioalm or the Application Insights Channel9 video series http://aka.ms/aic9

Summary

Historically Development, Test and Operations were very separate efforts with operational efforts like provisioning computers for load testing or collecting of production data while testing live services involved a lot of cross team effort. Visual Studio Online tools not only reduces this friction through shared tools– in cases such as Cloud Load Test actually reduces the effort needed, by dynamically provisioning and de-allocating load resources using Microsoft Azure. Checkout this FAQ to answer any questions you have about cloud-based load testing or if you need help troubleshooting.

*While Visual Studio Load Test can run Coded UI Tests.  It is not generally recommended and attempting to run a load test with a Coded UI Test from Visual Studio Online you will see it fails as the Visual Studio Online (Cloud) based test agent will not have access to your application’s user interface-which is how Coded UI Tests run.  The reason Coded UI tests are not generally recommended from a load test is the the UI will create a performance bottle neck i.e. issuing lots of commands to the single keyboard or mouse will either simply queue them or in the case of parallel runs mix up the commands causing erroneous results.  (Imagine a set of clicks not intended to be run in File Explorer sending a right click and another click bringing up the “format command”….)