Another in the ALM MVP Blog series. This article was written by Mike Douglas one of our most active ALM Rangers contributing to a bunch of the ALM Ranger projects!
Make sure and check out Mikes article: Continuous Feedback using Storyboarding, Code Reviews, and the Feedback Tool in Visual Studio 2012 ALM
(The ALM Ranger projects are projects that community members get together and build to give back to the community with a goal to provide with out-of-band solutions for missing features and guidance!)
Thanks for the great post Mike!
******************************************************************************
Code Coverage in Microsoft Test Manager by Mike Douglas
If you are QA professional, have you ever wondered how much of the application code is covered by your tests? Or if you are developer, have you ever wished you knew if there is any code that is no longer being called? Code Coverage for MTM makes it possible for QA professionals to know how much of the application code is being covered by Test Cases executed within MTM. Developers will know if all of code is being accessed by the application and a valid test case. Furthermore, developers can help the QA team identify scenarios in order to access the uncovered code blocks.
You might be asking how this is different than Unit Testing Code Coverage. Essentially, both measure and track how much of the code is being covered by tests. Unit Test code coverage is a good measure but doesn't necessarily measure the quality of the tests. Unit tests that cover a lot of code may still not be good tests. However, the code coverage that is captured when executing functional Test Cases and Exploratory testing in MTM will be valid functionality of the application. Therefore, if all of the application functionality and scenarios have be executed and there is still code that hasn't been tested, this code should be investigated if it’s valid.
Code Coverage for MTM is one of the features in Update 1 of Visual Studio 2012. In case you haven't heard, new features and bug fixes will be released on a Quarterly basis. This allows Microsoft to continue to improve the experience and respond to feedback without waiting for the next major release. This feature requires Update 1 to be installed on Visual Studio 2012/MTM, TFS 2012, and all of the Test Controllers/Test Agents.
Show me the coverage
Let’s jump in and look at the code coverage and assume we have the featured configured. I’ll show the setup in the following section. To begin, execute the Test Cases and Exploratory testing in Microsoft Test Manager (MTM) as you normally do. When you have completed the test run, you can view the results in a summary view or including details and code. As a QA professional, the most common way to view and track the code coverage will be through build selected in the test plan. Figure 1 shows the Build Definition Summary log. Highlighted is the additional section that will display when Code Coverage for MTM is configured. One of the useful features is that this section will continue to be updated as additional test cases and exploratory tests are executed and these numbers will reflect the current code coverage for all of the executed test cases and exploratory tests that have been run against the same build.
In the highlighted section in Figure 1, the first part explains that 4 separate test runs have been executed and are passing against this same build. In this case two DLLs have been instrumented and it shows the appropriate blocks of code being covered.
Figure 1 – Build Summary Log that keeps track of Code Coverage
If you want view the details of the coverage you can access this from two different locations. First if you want to see the overall results, click on the Coverage Results link as shown in the build summary log above. The second option is to open the test run and see the coverage for that particular test run. This may be useful to look back and see where you have increased the code coverage since the previous test run.
Figure 2 – Test Run Results with the Code Coverage results attached.
Both of these options will open the code coverage file within Visual Studio. This will allow you to drill into classes and methods. You are even able to open the code without the solution open to view the covered and non-covered blocks because the source symbols already had to be available to capture the coverage.
Figure 3 – Code Coverage Details in Visual Studio 2012
Enabling Code Coverage
Now that you have had a chance to see the Code Coverage for MTM in action, I’ll show you how to enable this for your test cases. Once you have Update 1 installed on your Visual Studio 2012/MTM, TFS 2012, Test Controllers, and Test Agent machines, the first thing to do is to create a Test Settings with Code Coverage collector enabled. Note: If you do not see the Web Server role, there must be an environment already configured and in ready state.
Figure 4 – Creating Test Settings with Code Coverage data adapter
To configure the Code Coverage Diagnostic Data Adapter, these are several settings to consider. The primary ones are:
· Specify alternative search paths for symbols (pdb) files (separated by semicolons). You can use ‘$(Build)’ macro in a symbol search path to specify the build number. In other words, if the PDB files are not deployed with the application DLLs in IIS, then you must specify the path to the PDB files.
· Specify a list of dll company name(s) to be excluded from data collection (separated by semicolon) – By default *microsoft* is excluded. However, in this example the company name for the FabrikamFiber sample application is Microsoft, so it was removed.
Figure 5 – Code Coverage Diagnostic Data Adapter Settings
The next step is to run a build definition that you want to use to track the code coverage. Typically you would want this build definition to build the assemblies that you want to test and have either an automatic or manual way to deploy them to the IIS server. Technically, as long as the PDBs are available for the assemblies on IIS, any build selected as the build is use can be used solely to track the Code Coverage.
Once the build has been completed, use this build as one of the inputs for your Test Run. To do this, select a Test Case to execute and choose Run with Options. The Run Options dialog will appear. Select the build from above as the Build in use and the right build configuration. Under Manual test runs, select the Test settings that was also created above. Finally, select the environment with the IIS Server.
Figure 6 – Test Case Run with Options Dialog
Run the tests with this option selected and it will begin capturing the code coverage. Alternatively, you can set these options as the default in the Test Plan so you don’t have to set them each time.
Figure 7 – Test Plan Settings for selecting options as default
Summary
Hopefully this gives you a good idea of what Code Coverage for MTM is and the problem it solves. I think this will be useful feature that keep relationships healthy between QA and Developers, measure test effort while performing manual or exploratory testing and increase effectiveness of test cases thereby improving the quality of the application and the testing effort.
Written by Mike Douglas
Mike Douglas is a solution consultant for Deliveron Consulting Services (www.deliveron.com) specializing in Visual Studio ALM consulting. Mike is experienced in helping development teams eliminate traditional silos between development, testing, and project management to establish cohesive processes with the Visual Studio ALM tools. He is also am an experienced presenter and has spoken at local user group meetings and conferences such as Heartland Developer Conference (HDC) and Kansas City Developer Conference (KCDC). Mike has worked with a number of technologies in his 14+ years of experience building enterprise solutions including BizTalk, SharePoint, SQL Server, and WCF. He actively maintains the open source project, Team Deploy, on CodePlex for deploying MSIs using Team Foundation Server. He also enjoys sharing his experiences in Visual Studio ALM and TFS, Code Generation, and CSLA.NET on his blog at www.CodeSmartNotHard.com.
Open Source Projects Mike actively contributes to:
- Team Deploy on CodePlex.com (http://www.codeplex.com/teamdeploy )
- ASP.NET CSS Templates on MSDN Code Gallery (http://code.msdn.microsoft.com/csstemplates )
- Automated Deployment MSI on MSDN Code Gallery (http://code.msdn.microsoft.com/msi )
Specialties
Visual Studio ALM, Team Foundation Server, Enterprise .NET Architecture and Design, CSLA.NET, Code Generation, BizTalk, Agile, Software Configuration Management.
