Leveraging Donovan’s great DevOps series on building his Ignite Demo here is a walk through for using Selenium with Visual Studio Team Services and the PartsUnlimited sample found at: https://github.com/Microsoft/PartsUnlimited 

 

Getting Started with Selenium in a Continuous Integration Pipeline

Performing user interface testing as part of the build process is a great way of detecting unexpected changes and need not be difficult. This walk through will get you started with using Selenium to Test your website in a Continuous Integration build.

You can use the the PartsUnlimited sample from Gitub running on your own computer or use the instance running at: http://partsunlimited.azurewebsites.net/

Getting Started:

1. Open PartsUnimited Solution from the 4.5 branch and Check in your solution to Visual Studio Team System (VSTS) or Team Foundation Server (this walk through will be assuming VSTS)
2. Add a new Unit Test project to the the PartsUnlimited Solution: Right Click on the solution and select Add>New>Project>Unit Test (Or you can can just do File>New>Project>Unit Test) .  For more information on this process please see: https://www.visualstudio.com/en-us/get-started/code/create-and-run-unit-tests-vs

4.  Add Selenium references for the drivers to your Unit Test Project. The drivers are how the browser is selected for executing the test.

Right-click on your Unit test project and select Manage NuGet Packages.   Add the following packages to your project

1. Selenium.webdriver.phantomJS.Xplatform //First as noticed I had to remove and re-add Selenium Webdriver if I installed it last 
2. Selenium.webdriver  //Gives you Firefox
3. Selenium.webdriver.Chromedriver
4. Selenium.webdriver.iedriver

 

 

(Note: You need to only install Chrome and IE drivers because the Firefox driver is part of the core framework and does not have to be installed separately.) 

 

5. To “Author” the Selenium Test, open the class file created for the Unit Test Project (by default called UnitTest1.cs) and Replace the contents of the file with the code below

 

namespace Partsunlimited.UITests

{
    using Microsoft.VisualStudio.TestTools.UnitTesting;
    using OpenQA.Selenium;
    using OpenQA.Selenium.Chrome;
    using OpenQA.Selenium.Firefox;
    using OpenQA.Selenium.IE;
    using OpenQA.Selenium.Remote;
    using OpenQA.Selenium.PhantomJS;
    using System;

    [TestClass]
    public class ChucksClass1
    {
        private string baseURL = "http://partsunlimited.azurewebsites.net/";
        private RemoteWebDriver driver;
        private string browser;
        public TestContext TestContext { get; set; }

        [TestMethod]
        [TestCategory("Selenium")]
        [Priority(19)]
        [Owner("FireFox")] //Using Owner as Category trait is not supported by the DTA Task
        public void RemoteSelenium()
        {
            DesiredCapabilities capability = DesiredCapabilities.Firefox();
            Uri server = new Uri("http://:80/wd/hub");
            this.driver = new RemoteWebDriver(server, capability);
            driver.Manage().Window.Maximize();
            driver.Navigate().GoToUrl(this.baseURL);
            driver.FindElementById("search-box").Clear();
            driver.FindElementById("search-box").SendKeys("tire");
            //do other Selenium things here! 
        } 

        [TestMethod]
        [TestCategory("Selenium")]
        [Priority(5)]
        [Owner("BuildSet")] //Using Owner as Category trait is not supported by the DTA Task
        public void TireSearch_Any()
        {
            driver.Manage().Window.Maximize();
            driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(30));
            driver.Navigate().GoToUrl(this.baseURL);
            driver.FindElementById("search-box").Clear();
            driver.FindElementById("search-box").SendKeys("tire");
            //do other Selenium things here!
        }

        ///

        /// Use TestCleanup to run code after each test has run
        ///

        [TestCleanup()]
        public void MyTestCleanup()
        {
            driver.Quit();
        }

        [TestInitialize()]
        public void MyTestInitialize()
        {   //Set the browswer from a build
            browser = this.TestContext.Properties["browser"] != null ? this.TestContext.Properties["browser"].ToString() : "firefox";
            switch (browser)
            {
                case "firefox":
                    driver = new FirefoxDriver();
                    break;
                case "chrome":
                    driver = new ChromeDriver();
                    break;
                case "ie":
                    driver = new InternetExplorerDriver();
                    break;
                case "PhantomJS":
                    driver = new PhantomJSDriver();
                    break;
                default:
                    driver = new PhantomJSDriver();
                    break;
            } 
           if (this.TestContext.Properties["Url"] != null) //Set URL from a build
            {
                this.baseURL = this.TestContext.Properties["Url"].ToString();
            }
            else
            {
                this.baseURL = "http://partsunlimited.azurewebsites.net/";   //default URL just to get started with
            }
        }
    }
}

At this point after build the project these Selenium Tests should run locally using the Test Explorer. 

 

But there are a couple of things that can go wrong:

1. Running IE Based Tests returns the error: OpenQA.Selenium.NoSuchElementException: Unable to find element with id == search-box

The IE security hasn’t been setup for Selenium or the driver has been installed correctly.  Please see: https://code.google.com/p/selenium/wiki/InternetExplorerDriver

2. JSON Errors when attempting a run any Selenium test…The PhantomJS driver wasn’t installed first.  To fix this simply remove the the Selenium.webdriver nuget package and re-add it. 

3. Tests run locally but on the build agent fails with the error:

Initialization method Partsunlimited.UITests.ChucksClass.MyTestInitialize threw exception. OpenQA.Selenium.WebDriverException: OpenQA.Selenium.WebDriverException: Cannot find Firefox binary in PATH or default install locations. Make sure Firefox is installed. OS appears to be: Vista.

Stack Trace:

at OpenQA.Selenium.Firefox.Internal.Executable..ctor(String userSpecifiedBinaryPath)

at OpenQA.Selenium.Firefox.FirefoxBinary..ctor(String pathToFirefoxBinary)

As the error suggests, Firefox doesn’t exist on the hosted build agent.  While you could run the Tests using the PhantomJS driver on the hosted build agent the other drivers will require to be run interactively –therefore will fail on the hosted build agent. For this example the best solution is probably running your build agent or using Selenium Grid to remote those calls…See the method RemoteSelenium() in the sample above and documentation on setting up a Grid Server. 

For directions on deploying PhantomJS.exe as part of your test please see: http://blogs.msdn.com/b/visualstudioalm/archive/2014/11/17/using-selenium-with-cloud-load-testing.aspx

It is worth, noting these Tests were written to be automation friendly and will look for the Browser type and a URL from the build definition.   This way we can pass in the desired URL during test execution during the release.  The reason the Test category trait was not used in the filtering is the Test Agent Task doesn’t support the category trait –yet.

 

6. Checkin the source and sync it back to the server. For more information on the checkin process please see the Sharing Code Walk through here: Create a build definition that builds your solution after each check-in, using continuous integration.  (will also walk through this below)

 

 

Create a build definition that builds your solution after each check-in, using continuous integration.

1. In your Visual Studio Team Services account that you checked in the sample of above we are going to Create a build definition that builds your solution after each check-in, using continuous integration. 
2. Navigate to the Builds hub of your of Visual Studio Team Services account and press the green action button to create a new build.

 

3.  Using the Visual Studio Build Template select the correct branch (if you created a branch for your sample), select continuous Integration and select “Create”

4. From the Visual Studio Build task, select the solution “Solution” option, open your version control (by selecting the ellipses) then select the  Partsunlimited.sln solution file.

Note as this is the only solution file in this sample this is not strictly necessary –but is still a good idea.

 

5. From the Visual Studio Test task set the URL, Browser and Priority parameters. Priority can be specified through the Test Filter criteria with no extra work.

The URL and Browser are passed in as Test Run Parameters.  If you look at the Unit Test class above you will see these parameters are used if they exist.

6. To make it easier to pass in the values for URL and Browser you could create variables and set the to Allow at Queue Time.

7.  Queuing the Build.  After you save the Build Definition you will be able to queue the build you created. Note the URL and Browser Properties can now be set in this dialog as a result of setting “Allow at Queue Time” in the variables created above.

Also the “hosted” build controller is not being used (this can be set as part of the build definition under the “General” tab) as the Selenium Tests will generally be run interactively-which would fail on the Hosted Build Controller.   For more information on running your own Build Agent please see:

https://msdn.microsoft.com/Library/vs/alm/Build/agents/windows

 

 

 

 

8. To validate the test results from a Build open the build results (clicking on the Build>Explore and you can see a snapshot of the Test Results in the build summary.

Additionally there is a Test results page that highlights the build-on-build changes including errors, stack traces and the ability to easily create a bug that contains this information.

Other related references:

• Creating and Running Unit tests: https://www.visualstudio.com/en-us/get-started/code/create-and-run-unit-tests-vs
• Using Selenium with cloud-based load testing: http://blogs.msdn.com/b/visualstudioalm/archive/2014/11/17/using-selenium-with-cloud-load-testing.aspx
• The getting started work through documentation:https://www.visualstudio.com/en-us/get-started/test/run-tests-with-builds
• Creating your Visual Studio Team Services Account: http://visualstudio.com
• Creating your own Visual Studio Team Services Build Agent: https://msdn.microsoft.com/Library/vs/alm/Build/agents/windows