Continuous Deployment: Dev/Test in Azure and Deploy to Production On-premises

Easily provision the required resources in Azure using the Azure Resource Manager templates, and deploy Web applications and SQL Server Databases to the Azure VMs using PowerShell scripts or PowerShell-DSC modules, and run tests that are automatically distributed across VMs using the Visual Studio Test Agent. Once the application's functionality is verified, deploy the application to on-premises machines using the same PowerShell scripts/DSC modules that were used to deploy the app to the Azure VMs, and by using the same tasks as such. The tasks provide an easy way to override the database connection strings, and any of the parameters that are changing between development, testing, staging and production. The PowerShell scripts (PowerShell 2.0 or 4.0, both supported) don't need to change at all, and all parameters can be easily overwritten in the tasks themselves. 

The blog explains how to setup a Build definition in the Build hub in VSO and configure tasks in the definition to dev/test in Azure and deploy to production on-premises.The Azure tasks described in the blog are in private preview on VSO, and if you would like to join the private preview then please send an email with your VSO's account name to devops_tools at Microsoft dot com.

So much effort it takes to develop, test and deploy an application successfully to Production. Is there a simpler way, is there some way to not write PowerShell scripts and just have tasks that can create IIS websites, deploy IIS Web applications, create App pools, deploy SQL Server databases, and create Windows Services? These not only have to be simple but also provide the full flexibility to cover any and all scenarios. Interestingly that is just as near as sending an email to devops_tools at Microsoft dot com. Interested? Hard to believe but we can deploy to anything, including SQL Clusters, SQL Always-On, Web Farms, Azure IaaS that are behind firewalls, and any of the Windows Service with any of the customizations. The how of this is just an email away and also a blog away.    

Azure SubscriptionThe

To deploy to Azure, an Azure subscription has to be linked to VSO using the Services tab in the Account Administration section.

Add the Azure subscription to use in the Build tasks by opening the Account Administration screen (gear icon on the top-right of the screen) and then click on the Services tab. Select Azure from the Add New Service Connection dropdown.

 Figure: Opening the Account Administration Screen.

Figure: Selecting an Azure connection from the Add New Service Connection.

Fill in the required details from the Azure account, and select credentials for authentication as certificates are not supported by Azure Resource Manager. The credentials have to be a work account because Microsoft accounts like joe@live.com or joe@hotmail.com are not supported.   

Figure: Adding the details of the Azure subscription. The data displayed in the dialog is not actual data and is for informational purpose only.

Build Agent

A build agent has to be deployed on a machine to build the code and to run the tasks. The build agent is xcopy deployable, and does not need any installation. The agent can be deployed on machines behind the Firewall and also on Azure VMs. The build agent uses the HTTPS protocol to communicate with VSO and can work across proxies and firewalls. VSO also provides Hosted build agents that can be used to build code and to deploy the application. The Hosted build agents are made available on-demand and run on Azure.

Deploy the Windows build agent as per the instructions in the Build docs. The following are some of the issues that one should keep in mind while deploying the build agent:

1. As described in the Build docs, either use the default queue or create a new queue.
2. Provide the rights to the account, under which the Build agent is running, in the VSO’s Control Panel named Agent Pools. This account should be added to the Agent Pool Service Accounts group.
3. Download and configure the agent by following the instructions.    
4. The tasks described in the blog needs Visual Studio 2015 and Azure PowerShell to be installed on the build agent. Visual Studio 2015 is needed for building the code and Azure PowerShell is needed to deploy Azure Resource Groups. Azure PowerShell can be easily downloaded and installed using the Web Platform Installer.  

Dev/Test Applications on Azure

Click on the Build Hub and then add the tasks as described below to deploy and test applications on Azure resources.

MSBuild/Visual Studio Build Task

The MSBuild/Visual Studio Build task is used to build the code.

Figure: MSBuild task in the Build Definition to build the solution.

Use the MSBuild or the Visual Studio Build task to build the app and test code. The Build documentation provides detailed instructions on how to build code. From this blog’s perspective, use the Source Code browser next to the parameter named Solution and select the solution file. Rest all parameters can be kept to their default values.

Azure Resource Group Deployment Task

The Azure Resource Group Deployment task is used to deploy Resource Groups in Azure using the Azure Resource Manager templates.

Figure: Azure Resource Group Deployment task in the Build Definition to deploy a Resource Group on Azure.

The different parameters of the task are as described below:

Azure Subscription: Select the Azure Subscription where the Resource Group will be deployed. This is a dropdown of the subscriptions that have been added in the Services tab.

Resource Group, Location: The name of the Resource Group and its location. If this is an existing Resource Group then the task will update the Resource Group with the Resources specified in the Azure template. If no Resource Group with the name exists in the Subscription then a new one will be created.

Template and its Parameters: The templates and the templates parameters file are the Azure templates available at GitHub and in the Azure gallery. To get started immediately, use this template that is available on GitHub. These files can be either checked in the Version Control of they can be part of the build itself. If the files are part of the Build, use the pre-defined system variables provided by the Build to specify their location. The variables to use are $(Build.Repository.LocalPath), if the templates are checked-in but are not built, or $(Agent.BuildDirectory), if the templates are built as part of the solution. Be sure to specify the full path like $(Build.Repository.LocalPath)\Azure Templates\AzureRGDeploy.json. Wildcards like **\*.json or **\*.param.json are also supported and there needs to be only file that matches the search pattern at the location. If more than one file matches the search pattern then the task will error out.

Override Template Parameters: The Override template parameters is used to override the parameters, like –storageAccountName azurerg –adminUsername $(vmusername) –azureKeyVaultName $(fabrikamFibre). 

Advanced Deployment Options: This section can be confusing if one does not understand the rationale behind the options. The following information will help in understanding the parameters and the proper use of them:

• Refresh SAS Token: Azure provides in-built ability to deploy applications using Resource Extensions (RE) for PowerShell-DSC or Custom Script Extension for any of the scripting languages for Linux (Python, Shell scripts). The scripts and application files need to be available on an Azure storage account because the RE downloads them to the VMs and then runs them. The storage accounts are normally secured, and a Shared Access Signature (SAS) token is needed to access them. More information about the Azure SAS token is here. This SAS token has an expiration date and the storage account cannot be accessed after the SAS token expires. This will break the Continuous Integration (Build) pipeline because the task will fail as it cannot access the storage account. The Refresh SAS token parameter refreshes the SAS token in the Azure template prior to sending it to Azure each time the Build is run. To do that it needs the name of the parameters for Azure Blob and SAS token as they have been specified in the Azure template parameters file. Multiple of these can be provided in Azure Blob Parameter and SAS Token Parameter by using the semi-colon delimiter like azurestore1; azurestore2; azurestore3. For example, the template parameters file has the following key-value pairs for Azure Storage account and SAS token. Then fill in azureStorage in the Azure Blob Parameter in the Azure Resource Deployment task and sasToken in the SAS Token Parameter.Note that Refresh SAS token is optional and only needed if deployment is being done to the Azure VMs using the Azure Resource Extension. As demonstrated in this blog, if the PowerShell on Target Machines task is being used to deploy the app then this Refresh SAS token is not needed.

"azureStorage": {
"value": "fabrikamfibre"
},
"sasToken": {
"value": " ?sv=2014-02-14&sr=c&sig=Dj1QOJups1%2Bf%2Beq989j%2FomowbmNS8Q2Qmf5eJEU9FGg%3D&st=2015-03-26T18%3A30%3A00Z&se=2015-04-03T18%3A30%3A00Z&sp=r"
}

• Virtual Machine Credentials: These are the Admin credentials for the Azure VMs that have been specified in the Azure template. The VM credentials are stored securely in the Deployment service, so that any subsequent tasks that need to access the VMs will not have to take this as an input. In this walkthrough, the tasks that need the credentials are Azure File Copy, PowerShell on Remote Machines, and Visual Studio Test Agent Deployment.  
• Test Certificate: The test certificate parameter requires background knowledge that is explained below. This parameter will be explained post the sections below.

Setting-up WinRM HTTPS on Azure VMs: Tasks like Azure File Copy, PowerShell on Target Machines, Visual Studio Test Agent Deployment run on the Build Agent machine and copy files or deploy apps to Azure VMs using the WinRM HTTPS protocol. For these tasks to work properly the WinRM HTTPS port (default port is 5986) needs to be opened and configured properly on the VMs. Opening the ports and configuring them with the certificates is done using the Azure templates. The sample template uploaded on GitHub shows how to enable the WinRM HTTPS port on Azure VMs and map them to a Public IP using the Azure resource provider’s wiz. Network Interfaces, Load Balancers and Virtual Machines. In addition, it also shows how to specify the Azure Key Vault and its secret, to download and install the certificate on the VM.

Azure Key Vault and Test Certificates: Azure Key Vault provides a secure way of storing certificates in Azure that can be easily downloaded and installed on the Azure VMs to enable WinRM HTTPS communication protocol. To create test certificates and to upload the certificates to Azure Key Vault follow the steps given below. Note that the Azure Key Vault can be in its own resource group and does not need to be in the resource group that is being dynamically created using the Azure Resource Group Deployment task. This way, once a certificate has been uploaded to the Azure Key Vault it can be reused across different Azure resource group deployments.

• Run the following commands from the developer command prompt to create the test certificate. Replace the Common Name (CN) with the Resource Group name and the password with the real password.

makecert -sv armtest.pvk -n "cn=*.westus.cloudapp.azure.com" armtest.cer -b 06/06/2015 -e 06/06/2016 -r
pvk2pfx -pvk armtest.pvk -spc armtest.cer -pfx armtest.pfx -po password

• For uploading the certificate to the Azure Key Vault follow the steps given below:
  • Download the KeyVaultUrl.ps1 available with this blog to the local disk.
  • Open Windows Azure PowerShell
  • Change directory to where the KeyVaultUrl.ps1 file was copied to.
  • Run the following commands:

 Add-AzureAccount 
 Select-AzureSubscription -SubscriptionName 
 Switch-AzureMode AzureResourceManager 
 Import-Module .\KeyVaultUrl.ps1 –Force

• To generate the secret Id run the command given below:

$a = Generate-KeyVaultUrl -resourceGroupName  -keyVaultName  -secretName  -location  -certificatePath  -password 

• Example:

$a = Generate-KeyVaultUrl -resourceGroupName nikhilnew13 -keyVaultName nikhilkeyvalut13 -secretName test -location "West US" -certificatePath "D:\cert\ArmTest.pfx" -password password

• Secret Id will get printed as well as get stored in $a variable and it will be similar to – https://fabrikamvalut.vault.azure.net:443/secrets/fabrikam/256892c857714ec0bd76529147cf6b11

Test Certificate: The Build Agent uses the WinRM_HTTPS communication protocol to deploy apps to the Azure VMs. For secure communication, certificates are used and they need to be installed on the VMs. For developing and testing apps, usually test certificates are used, and by checking-off the Test Certificate parameter, the trusted certificate authority (CA) validation is skipped. Note that the Common Name (CN) of the certificate has to be same as that of the Fully Qualified Domain Name (FQDN) of the VM. If the CN and FQDN different then the task will error out.    

Azure File Copy

The task is used to copy application files and other artifacts that are required to install the application on Azure VMs like PowerShell scripts, PowerShell-DSC modules etc.

Figure: Azure File Copy for copying files to Azure blobs or Azure VMs.

The task provides the ability to copy files to an Azure blob or directly to Azure VMs. Even when the target is Azure VMs, Azure blobs are used as an intermediary and the files are copied to it first and then downloaded to the VMs. The tasks uses AzCopy,
the command-line utility built for fast copying of data from and into Azure storage accounts. The different parameters for copying files to Azure VMs are as described below:

Azure Subscription: The name of Azure subscription, where the Azure storage account is located. The storage account is accessed using the stored credentials of the Azure account in the Services tab.

Source: The source of the files. As described above using pre-defined system variables like $(Build.Repository.LocalPath) make it easy to specify the location of the build on the Build Automation Agent machine. The variables resolve to the working folder on the agent machine, when the task is run on it. Wild cards like **\*.zip are supported and should resolve to a single file or a folder.

Storage Account: The name of an existing storage account in the Azure Subscription specified earlier.

Destination: The target for copying the files and is either an Azure blob or VMs. The section below details the parameters that need to be filled-out if the target is Azure VMs. Note that for copying the files to VMs, they are first copied to an automatically generated container in the Azure storage account, and then from there to the VMs. The container is deleted after the files are copied successfully to the VMs.

• Resource Group: Name of the resource group that contains the Azure VMs.
• Select Machines By: The parameter is used to copy the files to a subset of VMs and the subset can be specified by the names of the machines or the tags on them.  
• Copy to Machines: If copying to a subset of machines provide a comma separated list of the VMs here, else if using tags then provide the tags in the format Role:Web; OS:win7. The default is to copy to all the VMs in the Resource Group.
• Destination Folder: The folder in the Azure VMs where the files will be copied to. Environment variables are also supported like $env:windir, $env:systemroot etc. An example of the destination folder is $env:windir\FabrikamFibre\Web.
• Clean Target: Checking this option will clean the destination folder prior to copying the files to it.
• Copy Files in Parallel: Checking this option will copy files to all the VMs in the Resource Group in-parallel, hence speeding up the process of copying.

Destination: If the target is Azure blob then the following parameters need to be filled out.

• Container Name: The name of the container where the files will be copied to. If the container does not exist then a new one will be created with the name provided in this parameter.
• Blob Prefix: A prefix for the Blobs that can be used to filter the blobs like appending the Build number to the blobs, so that all the blobs with the same build number can be downloaded from the Container.

PowerShell on Target Machines

The task is used run PowerShell on the target machines. The task can run both PowerShell scripts and PowerShell-DSC scripts. For PowerShell scripts, PowerShell 2.0 is needed on the machines and for PowerShell-DSC scripts Windows Management Framework 4.0 needs to be installed on the machines. WMF 4.0 ships in-the-box in Windows 8.1 and Windows Server 20012 R2.

Figure: PowerShell on Target Machines task runs on the machines to install applications

The different parameters of the task are explained below:

• Machine Group: The name of the Azure Resource Group.
• Select Machines By: The parameter is used to specify the subset of VMs, where the PowerShell will be run and the subset can be specified by the names of the machines or the tags on them.  
• Deploy to Machines: If deploying to a subset of machines provide a comma separated list of the VMs here, else if using tags then provide the tags in the format Role:Web; OS:win7. The default is to deploy to all the VMs in the Resource Group.
• PowerShell Script: The location of the PowerShell scripton the target machine like c:\FabrikamFibre\Web. Environment variables can be also used like %systemdrive%\Web or %temp%\ FabrikamFibre\Web
  etc.
• Script Arguments: The arguments needed by the script, if any provided in the following format -applicationPath $(applicationPath) -username $(vmusername) -password $(vmpassword).
• Initialization Script: The location of the data script that is used by PowerShell-DSC and the location has to be on the target machine. It is advisable to sue arguments in place of the initialization script. 
• Advanced Options: The advanced options provide more fine-grained control on the deployment.
• Run PowerShell in Parallel: Checking this option will execute the PowerShell in-parallel on all VMs in the Resource Group.
• Session Variables: Used for setting-up the session variables for the PowerShell scripts and the input is a comma separated list like $varx=valuex, $vary=valuey. This is mostly used for backward compatibility with the earlier versions of Release Management product and it is advisable to use arguments in place of the session variables.     

Visual Studio Test Agent Deployment

The task is used to deploy the Visual Studio Test Agent on the machines. The task runs on the Build Agent machine and uses PowerShell to deploy the Test Agent to the machines.

Figure: Visual Studio Test Agent Deployment task runs on the machines to install and configure the test agent.

The different parameters of the task are explained below:

• Machine Group: The name of the Azure Resource Group.
• Select Machines By: The parameter is used to specify the subset of VMs, where the test agent will be installed, and the subset can be specified by the names of the machines or the tags on them.  
• Deploy to Machines: If deploying to a subset of machines provide a comma separated list of the VMs here, else if using tags then provide the tags in the format Role:Web; OS:win7. The default is to deploy to all the VMs in the Resource Group.
• Username: The credentials that the test agent will use to run on the machine.
• Password: The password of the user.
• Interactive Process: Checking this option will configure the test agent to run as an interactive process and this is needed for running the Coded UI tests.
• Test Agent Location: The test agent is downloaded and installed on the machines from http://go.microsoft.com/fwlink/?LinkId=536423. The default behavior can be overridden by manually downloading the test agent and then providing the test agent’s location as a UNC or local path.
• Update Test Agent: Checking this option will update the test agent and its configuration.
• Enable Data Collection Only: Checking this option will configure the test agent to only collect data, and no tests can be run using the test agent. Useful when data needs to be collected from the application-under-test. 

Visual Studio Test using Test Agent

The task is used to run tests on the machines where the test agent has been deployed.

Figure: Visual Studio Test using Test task runs tests on the machines where the test agent has been deployed.

The different parameters of the task are explained below:

• Test Machine Group: The name of the Azure Resource Group where the tests will be run. The test agent should have been already deployed and configured on the VMs using the Visual Studio Test Agent Deployment task.
• Test Drop Location: The folder on the VMs where the test binaries are located. Environment variables can be also used like %systemdrive%\Tests or %temp%\Tests etc.
• Test Assembly: The test binaries for running the tests. Wild cards can be used like **\*FabrikamTests*.dll that will use all test assemblies with FabrikamTests in their name.
• Test Filter Criteria: Used to specify the test filter criteria like ‘Owner=Tom&Priority=0’.
• Platform: The platform against which the tests will be reported like any cpu or x86 or x64.
• Configuration: The configuration against which the tests will be reported like any debug or release.
• Run Settings File: The location of the run setting file on the Build Automation Agent machine. Predefined variables like $(Build.Repository.LocalPath) can be also used.
• Override Test Run Parameters: Used to override parameters in the TestRunParameters section of the run setting file like WebAppURL=$(appURL);Port=8080. Here the $(appURL) is a variable that has been defined in the Variables tab of the Definition.
• Test Configurations: Used to associate a test case filter against a test configuration ID. The syantax is :; DefaultTestConfiguration:, like FullyQualifiedName~Chrome:12.
• Code Coverage Enabled: Checking this option will enable code coverage during the testing.
• Application Under Test Machine Group: The name of resource group where the Application under Test is running. If the same resource group is being used for deploying applications and running tests then the name of Test Machine Group and Application Under Test Machine Group will be same.   

Machine Group Actions

The task is used to start/stop/restart/delete Azure Resource Groups.

Figure: Machine Group Actions task to start/stop/restart/delete Azure Resource Groups.

The different parameters of the task are explained below:

• Azure Subscription: The Azure Subscription, where the resource group is located.
• Machine Group: The name of the Azure resource group.
• Action: The action to be performed on the resource group wiz. start, stop, restart, or delete. The block and unblock actions currently do not work with Azure resource groups and once that has been fixed, the updates will be posted to the blog.
• Select Machines By: The parameter is used to specify the subset of VMs on which the action will be performed, and the subset can be specified by the names of the machines or the tags on them.  
• Apply to Machines: If deploying to a subset of machines provide a comma separated list of the VMs here, else if using tags then provide the tags in the format Role:Web; OS:win7. The default is to deploy to all the VMs in the Resource Group.