I had earlier blogged about how we use Release Management to run our test automation in the RM team. Since then, RMO has added support for parallel environments (with the S94 payload), and we have tweaked our automation pipeline to leverage this feature.
We now have all our test environments in a single RD, and the deployment condition (which you can bring up by clicking “…” on the Environment –> Deployment conditions) for all of them is “After release creation”.
This gives us much nicer view for monitoring how stable the builds have been over time.
In addition, we also use release promotion to test our AtDtMismatch scenario. Specifically, the AtDtMismatch environment tests the scenario of the application tier (AT) being at a different version from the data tier (DT) e.g. AT could be at M97 and DT could be at M96.
In the AtDtMismatch environment, we install SPS. TFS and RMO from the previous sprint e.g. M96, run the RmEqTfs test suite, then upgrade the RMO AT only to M97, run the RmGtTfs test suite, and finally upgrade the RMO DT to M97, and again run the RmGtTfs test suite. It therefore makes sense to promote a release to the AtDtMismatch environment only if the basic RmEqTfs, RmGtTfs and dev-fabric upgrade environments are green. This can now be achieved by clicking “…” on the AtDtMismatch Environment –> Deployment conditions –> After successful deployment on another environment..
At runtime, the AtDtMismatch environment gets deployed only when RmEqTfs, RmGtTfs and DevFabricUpgrade are green
You can get a nice overview of the quality of your build in the Test tab of the Release: Open the Release –> Test –> select “All” in the environment dropdown –> select “All” in the Outcome dropdown –> click ‘-‘ (Collapse All).
You can then drill down into the failed tests to see when they began failing etc.
Converting multiple RDs into a single RD with multiple environments
I thought I should write a bit about the process we used to convert the 9 RDs RM.CDP.* into a single RD RM.CDP with 9 environments (with 2 more environments – MmsLoadTest and AtDtMismatch – into which we promote the release as required). Lets take the example of the RM.CDP.TfsOnPrem RD being converted into an environment in the new RD:
1) In RM.CDP.TfsOnPrem, save the environment as an environment template:
2) Change the trigger from Continuous Deployment to Manual for RM.CDP.TfsOnPrem since we want to effectively deactivate it:
3) Add an environment into RM.CDP based on this template: Edit RM.CDP –> Add Environment –> select “Custom” tab of Deployment Templates –> select the template that you just saved off.
4) So far, so good. Now the painful part: You need to manually ensure that all the variables that were present in RM.CDP.TfsOnPrem are also available in RM.CDP. We just trundled through all the variable of RM.CDP.TfsOnPrem and added them to RM.CDP. Variables that had pretty much the same value across all the test environments ended up Release-level variables, while variables that had different values for each test environment ended up being Environment-level variables.
Here is a view of the Release-level variables of RM.CDP:
Here’s a view of the Environment-level variables of RM.CDP:
A quick note on how to add Environment-level variables. Its not through the Configuration tab, but rather through the Environment option i.e. Click “…” on the environment –> Configure variables:
5) Finally, we renamed RM.CDP.TfsOnPrem to RM.LegacyCDP.TfsOnPrem, and we don’t use it any more. After a few months of disuse, we expect to delete it.
Looking forward
The next feature that I am really looking forward to is support for branch filters in RMO, so that we can easily find the quality of a build in a particular branch. To get this right, we need to normalize the branch data that is stored in the DB and then expose it in the UI. Hopefully we will get this feature out in a few sprints, at which time I will blog again on how we use it.