Part 2–Using Traits with different test frameworks in the Unit Test Explorer

As promised one my the ALM MVPs, Terje Sandstrom, has delivered his Part 2 of his Unit Test Series: “Using Traits with different test frameworks in the Unit Test Explorer”.

From the traffic and comments on his first post these are clearly very appreciated articles!

Thanks Terje!

*********************************************************************************************************************************************’

Part 2:  Part 2–Using Traits with different test frameworks in the Unit Test Explorerby Terje Sandstrom

In Part 1  traits in general was described, and we used examples from using MSTest as a testing framework.  In this part we will built upon what we did in Part 1, but extend it with many of the other frameworks.  The Unit Test Explorer is test framework agnostic!  It works with any framework, as long as there exist an adapter, and such adapters are coming for NUnit, XUnit, MBUnit, Javascript-Chutzpah and even more.  A list of these and links to information and downloads for this can be found inthis blog post. Note that some of these are being updated along with the release of Update 1. See summary at the end of the blog post.

The summary at the end also shows the syntax for the different traits in the different test frameworks.

We will continue with the code examples from Part 1, and do something that perhaps is not so usual, mixing in some of the other frameworks and let them play together !  That way it is also easier to see the differences between these.  It will be done step by step, so the examples expand as we add more.  The sequence is completely random and doesn’t express any preferences.

What is pretty cool, is that we show that in principle all types of tests can be run at the same time, many even in the same project, although Managed C++, Native C++ and JavaScript tests stay in their own projects, but all in the same solution, and the results will be merged together in the Test Explorer !

The different frameworks and their adapters are at somewhat different levels regarding how compliant they are to the new Trait system.  Information is given for each framework on what is missing, and also expected time of arrival for the improvements where we have managed to obtain that information.

Adding in NUnit

Now let us mix in a NUnit Test first. In order to do this you must install the NUnit Test Adapter from Visual Studio Gallery, and you must add a NUnit package from NuGet to your project. Adding the NuGet is done in a few steps as shown in the image series below.

Step A: From Visual Studio Tools menu, choose Library Package Manager/Manage NuGet Packages for Solution.

Step B: If this is the first time you add this to any solution, choose Online, and add “NUnit” in the search box, select the NUnit. (Note, you don’t need the NUnit.Runners, as VIsual Studio has its own runner that will run these for you). If you already have done this once, go instead to the Installed Packages, NUnit will be listed there and press the button “Manage”. Any of these will take you to step ‘C’.

Step C: Choose the test projects where you want to include NUnit.

Your project will then look like this, where (1) is included Dll and the NuGet package configuration file. (If you ever want to remove it, these two files are the ones to remove. If you got more packages, you must open and edit the packages.config.)

Now you can add a new NUnit test file to your project, 2) in the screenshot above.

NUnit has a different way of setting up categories, in fact it can be done in two different ways. You can add the category as a text string, the same way as you do it in MSTest above, or you can derive a new class from the Unit CategoryAttribute class. We have done both things below:

When opening up the Test Explorer now we see the following:

Both methods are in the same Trait, named Category [CI], even if they were defined differently. That is pretty cool !

Notice that the MSTest test that is in the CI category is placed in its own group.  The categories from NUnit and MSTest are different even if they are spelled equally.  That, however, only applies for categories.

If we add in another test method and decorate that with the same kind of attributes that we did in the MSTest example in Part 1:

In NUnit there are no explicit Owner and Priority property, but you can use the general Property attribute instead.  The NUnit Property is more advanced than the MSTest Property, it has overloads for string, int and double, so implementing Priority with this makes it equal to the MSTest specific Priority.

The output from this then becomes:

Note that except for the categories, all the other properties are collected together for both MSTest and NUnit.

Status

1. Current version of the adapter at time of writing is 0.93.  There is a bug concerning interoperability with Native C++ which is fixed in the upcoming 0.94 version, ETA Nov 30th 2012.
2. Version 0.94 will also add class based Categories, see discussion below under xUnit. 

Adding in xUnit.net

The xUnit.Net adapter can also be found at Visual Studio Gallery (traits implemented in version 0.9.3), and the framework is found on NuGet the same way, search for “xUnit”, and install both the same way as done for NUnit, adding the libraries to the test project. 

In the xUnit.net framework, the different traits are implemented by decorating the test method with the Trait attribute.  This is used for all kind of properties.  Everything is a Trait in xUnit.  It doesn’t have any int overloads, so the priority must use a string representation, but the Test Explorer will treat that string and the MSTest int as the same.

The same code then becomes:

You can also derive new classes from the TraitAttribute class, so if you like you can implement for example the Category, specific categories you want, and the Priority attributes that way:

The results with both NUnit, MSTest and XUnit running at the same time is then:

Both the NUnit and XUnit categores fall sweetly into the Categories group, MSTest are in its own special category types (CI, Developer and Production).  They all treat the other traits equally. Notice that the special new attributes made above for ‘CI’ and ‘Priority’ are treated the same as the original ones.

Class based attributes

When I write unit tests I often group them by class.  I rarely put two different types of unit tests in the same class.  So, what I would like is to be able to decorate the class with Traits too.  Those traits should then be treated as applying to all the methods in that class.  It turns out that XUnit is the only framework I have tested so far that have this capability.  MSTest refuses you to add the Traits to the class, they are bound to methods only.  NUnit allows you to do it, but these traits have no effect yet, but the next version of the NUnit adapter should have this capability too for categories.

We add some new test classes to all the projects, looking like this:

(Equally for NUnit and MSTest, except different suffices, N11/N12 and Ms11/Ms12.

And the results show that the NUnit and MSTest ends up the No Traits group, whereas the XUnit comes in correctly

Adding in MbUnit

To add in MbUnit one need to add first the Gallio framework, then the MbUnit adapter, and finally using NuGet to get the Gallio & MbUnit dll’s into your project:

MbUnit has specific Traits, which matches some of the others, but not all.  It has Category, Author and Importance.

MBUnit also allows for derivation of new attribute classes like NUnit and xUnit.net, but the adapter doesn’t support this for traits.

One important thing to notice is that the adapter don’t accept the Importance attribute shown above. 

The current version of Gallio/MbUnit, version 3.4 does not work correctly with its own version 3.4 of VS Test Adapter, giving an exception when you try to run it. The adapter seems to have been built for an earlier version.

I did download their adapter code and made a few corrections and added the Trait code, just to see how the underlying MbUnit/Gallio treats this.  The adapter is only a bridge between the test runner and the test framework.  I will try to get these code changes back up to the MbUnit project, so it should be ready to go in the next release.

Notice that we now are running 3 adapters at the same time - cool !

Status

Current version of the adapter (3.3) does not support Traits, but next version of the adapter will do so.

Adding in JavaScript testing with Chutzpah

Chutzpah is the test adapter for the QUnit and Jasmine test frameworks for JavaScript.

QUnit and Jasmine have the keywords ’Module’ and ‘Group’ respectively that can be used to decorate the test methods.  Those are the only ones available who can be used to implement Traits.  They are semantically not meant like Traits, but can be used that way.

Also note that Chutzpah currently runs tests per test file, not per test.  So if you select a single test and runs that, all the other tests in the same file are also being executed. This imply you should consider how your Modules and tests are organized by file.  One file should not contain more than one Module, if you intend to run them separately. 

Again, I added a few lines trait coding to the Chutzpah source code, and this is how it looks, using Module as the keyword for a Trait.

And the results, together with the other tests.  In the screenshot below, I have chosen to execute the first test in group a, and as can be seen the other tests are also run.

Status

1. The current version at time of writing (Nov. 12th 2012) of Chutzpah is 2.2.1. This version does not support Traits.  The next version will support traits.

Summary

Release schedules for trait aware 3rd party framework adapters

Traits summarized

The first column merely sets a generic name for the traits, to be able to compare them between the adapters/frameworks. The ‘----‘ means “Not implemented”.

Property is a generic property/value set, and support for that is very nice to have. Implementing that means the traits are very extensible, and you can twist them to anything you like, and have full support of that in the Test Explorer.

Have fun !

******************************

By

Terje Sandstrom

------ Chief Software Geek at Inmeta Consulting in Scandinavia ----- and a Visual Studio ALM MVP