Visual Studio Team Services – Testing Tools Roadmap

Over the last few months, there has been a consistent ask for Visual Studio Team Services (VSTS) / Team Foundation Server (TFS) Testing Tools roadmap.  I would like to share our strategic direction, what we have delivered over the last 6 months and where we are headed.  It would be great to get your feedback on what we are doing right and where we are missing the boat …

Given various trends (Agile, DevOps, Cloud, Mobile) in application development, here are the key shifts we see in testing:

• Businesses have moved to faster releases– time to market has become a key differentiator.  Multi-channel applications that work on web, phones, PCs and tablets, combined with support for Bring Your Own Device (BYOD) in enterprises, are leading to test configuration explosion.  In terms of quality, expectations have increased in all dimensions: increased customer value, shortened release cycle, more test configurations, moderate budgets and high business risks associated with lapses in quality.
• Engineering teams have become efficient in making incremental changes and doing Continuous Integration (CI).  Developers are embracing the notion of Test early, Test often. They are writing more unit tests and executing them frequently as part of CI process.  They are adopting DevOps and automating the BDT (Build-Deploy-Test) workflow for one or more environments (Dev/QA/Pre-Prod/Prod) and executing automated tests appropriate for each environment.
• Manual testingis still relevant.  There are more folks manually testing the product than you realize: feature teams do a bug bash, product managers play with the product, business users/domain experts do User Acceptance Testing, early adopters use the product in real world scenarios etc. Everyone wants to provide feedback and improve quality of the product, but does not have necessary tools to do so in a friction free manner.
• Enterprises are keen to optimize resourcesby leveraging cloud based infrastructure. As applications move to cloud, security and performance testing becomes a critical need.  Test execution is moving to cloud and there are many SaaS offerings for load and performance testing, mobile device testing, that enterprises are leveraging.

With that context, here are our focused areas of investments:

• Dev Inner Loop – Unit Testing in Visual Studio IDE
• Testing in Continuous Integration (CI) and Release Management (RM) or Continuous Delivery
• Manual and exploratory Testing
• Load and Performance Testing
• Integration with 3^rd party test services

We have been executing on this strategy for the last 6 months.  Here is the key value we have delivered so far to Visual Studio Team Services and Team Foundation Server 2015, Update 1 and Update 2 (RC):

Dev Inner Loop – Unit Testing in Visual Studio IDE

In Visual Studio we have an open and extensible test platform with tests being written using various test frameworks and executed using a variety of adapters. We will continue to make feature enhancements that benefit all Unit Testing frameworks – MSTest, Junit, XUnit, NUnit etc. There has been a consistent ask for efficient execution of unit tests – we have enabled support for Parallel execution leveraging multiple cores and Incremental validation that runs only impacted tests based on product changes.

MSTest– We will continue to support and enhance MSTest. For ASP.Net v5 applications we have enabled support for writing MSTest based tests and collecting code coverage data.   We intend to address other high user voice asks around data row and test category support at class/assembly level etc.

What we have delivered:

• Parallel test execution leveraging available cores on the machine.

• Incremental validation (aka Context sensitive test execution) – detect and execute tests only from updated test containers.
• MSTest framework support and code coverage for ASP.NET 5 applications
• NuGet support for MSTest framework components

What’s coming next:

• MSTest enhancements – Data row and Test category support
• Coded UI support for Edge browser
• Test explorer enhancements – Test assembly discovery in multiple paths, Rich filter, sort and group by capabilities etc.

Testing in Continuous Integration (CI)

Running and analyzing tests is an integral function of a great CI system. Our goal is to provide a great experience for in-context test and code coverage results with actionable insights for any application. We are planning to enhance this experience on a continuous basis and add more value, charts and trends as we go along.

What we have delivered:

• An aggregate summary of test results across all test runs in the build.
• In-context Test results with actionable insights. Tests failing for the first time in the bug are flagged as New failures to help you identify regressions. For tests that continue to fail across multiple builds, you can see how stale the failures have become and navigate to the build in which a failure was introduced.

• Support for any UT framework of your choice – MSTest, XUnit, NUnit, Junit, etc.
• Added support for xUnit, NUnit, Junit and the .trx file format to the Publish Test Results task.
• Testing Java Applications:
  • Support for JUnit: You can upload JUnit results to TFS from Ant, Gradle, and Maven tasks.  You can also run tests using a different command line tool and upload results using Publish test results task.
  • Code Coverage support for Java applications: Choose between Jacoco and Cobertura as Code Coverage tools for your Java project. The build file is modified on the fly to use the correct tool and the results are uploaded to TFS.
  • Publish Code Coverage task supports Jacoco and Cobertura: If your build file is already set to collect code coverage information, or if you’re using a different Code Coverage tool (such as Istanbul) that supports outputting results in Jacoco or Cobertura format, you can use the Publish Code Coverage task to upload code coverage data to TFS.
• Trend charts that show the count of failed tests and test duration for the last 10 builds. You can add these charts to the dashboard.

Testing in Release Management (RM)

Similar to CI, test results will be surfaced for each environment (ex. Dev/QA/Prod) of a Release. You will be able to assess the quality of a release in a particular environment and make an informed decision about moving to next environment in your release pipeline. Given the large number of tests you will run in the context of a release, we will focus on efficient execution of Functional UI Tests, Load/Performance Tests etc. We will enable seamless execution on any cloud infrastructure – SCVMM, VMWare, Azure and support hybrid configurations for your applications and test rig.

What we have delivered:

• In-context test results in RM for each environment – similar to Build
• Parallel Test execution on multiple machines
• Support for Coded UI and Selenium for Functional UI Testing
• Support for Standard environments and Azure VMs in Build-Deploy-Test (BDT) workflow

• MTM Test Plan and Test Suite execution improvements in CI and RM

• Support for VMWare and SCVMM environments in BDT workflow

What’s coming next:

• Configurable Test Widgets on Dashboards
• Test history across branches (CI) and environments (RM)
• Test Impact in CI

Manual and Exploratory Testing

We are focused on empowering everyone on the team to provide feedback and improve quality.  We have delivered a new exploratory testing experience for web and mobile applications.  Learn about exploratory testing on the web and get started by installing the Chrome browser plug-in from the Marketplace.

We intend to deliver a compelling web experience for all manual and exploratory testing needs.  We want to enable testers to be more integrated into the Agile practices – easily add tests for a user story, update test results inline on Kanban boards, and provide a quality perspective during standups.

Given our focus on cross-platform, Agile + Test scenarios, integration with 3rd party test services, end-to-end traceability we have decided to deliver most of the new value on our web experience, i.e., Test hub.   We are aggressively closing feature gaps between Microsoft Test Manager (MTM) and our web experience (Test hub).  We will continue to support MTM but we encourage users to use our web experiences and benefit from all the new value we are creating.

What we have delivered:

Manual Testing

• Export test outcomes, Manual Test iteration results
• Retention Policy for test results
• Filter test plans, Delete Test plans
• View tests from child suites

Exploratory testing (XT) on the web

• Simplified screenshot and note captures
• File bugs from your exploratory testing session. The notes, annotated screenshots, team area and iteration paths, and system and browser information are automatically captured for you.
• Search and update existing bugs
• Integration with Perfecto Mobile for testing Mobile applications
• Standalone and Connected (to TFS or VSTS) mode
• Create tasks for issues found in your XT session.
• Explore work items – Search for a work item from within the XT session and then associate it with the session. You can then refer to the acceptance criteria/description during your exploratory testing. You get the benefit of end–to–end traceability between any bugs or tasks you file during your exploratory session and the selected work item.

What’s coming next:

• Inline tests directly on Kanban board
• Configuration management in web access
• Test Plan/ Test Suite clone (via marketplace extensions)
• Offline Testing (via marketplace extensions)
• XT-Image action log, traceability & video capture
• XT-Insights across exploratory testing sessions
• XT-Support for stakeholder solicited/unsolicited workflows

Load and Performance Testing

Our intent is to offer a compelling load testing service that enables you to create a load test rig in minutes leveraging multiple Azure data centers, and provides quick insight into your application performance in real world scenarios.

What we have delivered:

• JMeter support
• Web experience for Author-Execute-Analyze load tests
• Integrated Performance Test experience for Azure Web Applications,
• Ability to run load tests as part of CI/RM pipeline

What’s coming next:

We are actively discussing roadmap for load testing and I will share more details later.

Integration with 3^rd party test services

We have enabled integration with few 3^rd party tests services as extensions.  We will continue to integrate more 3^rd party services that offer compelling value to our customers.

One more thing – we are committed to having a great DevOps story for Mobile Apps.  With VSTS and HockeyApp integration, you have an ability to build your Android, iOS or Windows application, deploy to a device in-house or managed by a 3^rd party device cloud provider, run automated tests, perform exploratory testing and then leverage HockeyApp service to distribute to beta users, collect live crash reports, get feedback from real users and analyze test coverage!

Let me clarify our positioning on a few other areas that are important:

Microsoft Test Manager (MTM) and Lab Management (LM)

There are many customers who use MTM and Lab Management for Manual, Exploratory, and Automated Testing.  For manual and exploratory testing, we have been delivering all new value on the web experience – Test hub.  We intend to move the remaining experience in MTM to the web soon.

For automated testing (BDT) scenarios we had critical gaps at TFS 2015 RTM in Test Plan/Test Suite based execution, On demand execution, Associate automation , support for SC-VMM environments etc.  We are aggressively closing these gaps and adding new features such as VMware and Azure environment support, simplified application deployment etc. This should enable customers to move forward to our new Build/RM workflow.

We are not making any enhancements to existing MTM and lab management product.  Given our focus on cross-platform, Agile + Test scenarios, integration with 3rd party test services, end-to-end traceability, simplified Build/RM workflow for BDT,  we have decided to deliver most of the new value on our web experience, i.e., Test hub.  We will continue to support MTM but we encourage users to use our web experiences and benefit from all the new value we are creating.

Coded UI and Selenium

Coded UI continues to be our preferred tool for Functional UI Automation for Desktop, Web and Store Apps.  With WebDriver becoming a W3C standard and strong enterprise demand to support Selenium for Web Apps, we will be improving our support for authoring and executing WebDriver based Selenium and Appium tests.  We will do selective feature enhancements to Coded UI such as UWP, Edge suppor etc. and encourage customers to use Selenium and Appium for Functional UI testing.

For more detailed information and getting started please refer to the blogs below:

• Test results in Build/CI
• Testing Java Applications
• Testing on standard environments and Azure VMs
• Testing web applications using Selenium
• Test dashboards in Build/CI
• Parallel Test execution
• Incremental validation
• Exploratory Testing
• Unit testing and Coded UI support for UWP applications

We are keen to hear from you … Appreciate you taking time to read this rather long blog post.

If you like the features you can get started on using testing tools on visual studio team services.

If you have feedback or feature requests, please do leave us comments.

Anand Kamat