Using advanced diagnostics to debug issues in Cloud Load Test.

Cloud Load Testing can be used for performance and load testing of an application by generating load from Azure.

It is often observed that debugging on cloud is a little difficult as the machines are not directly accessible to the user. It is not possible to collect logs or any other information from the test agents easily.

With advanced agent diagnostics feature customer will be able to collect logs from the test agents and download it from an Azure Blob and analyze the data.

Some scenarios where this feature is useful:

1. My unit tests are failing on cloud and the test logs in Load test results  are not sufficient to debug the issue. In this scenario I can put trace statements in my unit-test for logging and analyze the UT failure with these trace logs.
2. My test generates some files which are needed to understand the test output better. You can collect these files by just copying the files to the "Result Directory".
3. If I want to have detailed diagnostic to find networking bottlenecks using packet capture tools such as Wireshark.

 You can install WireShark as part of the Setup Script (Link ), redirect the logs to "Result Directory". This feature will upload the files to blob from where I will be able to access them.

How to use trace logs in CLT for debugging ?

Using trace logs is as simple as writing any trace statement in your code.

Eg:

Trace.TraceInformation("Information :{0}", DateTime.UtcNow);

Trace.TraceWarning("Warning :{0}", DateTime.UtcNow);

Trace.TraceError("Error: {0}", DateTime.UtcNow);

These trace statements can be used in your UnitTest code, Load Test Plugin, WebTestPlugin code.

The trace logs will be available at the end of the run from the blob.

TraceFileName: CustomTraceListener.log

 Right now we have enabled the trace logs at warning level.

You can contact the team if you want to get the level reset to some lower or higher value at vsoloadtest@microsoft.com.

How to collect other files from test agent?

We have enabled files upload from testagent to Azure blob from a specific directory(ResultDirectory) on agent disk space.

The ResultDirectory is an environment variable which can be used in code/setup script/cleanup scripts.

You can drop the files you need from test agent to this directory and download it from blob.

These files will be available from blob as and when they are available on agent(with some latency)

How can you drop the files to the Result directory?

1. You can do this in any of the places you have code, in webtest plugin, load test plugin or unittest.
2. You can do this in setup script.
3. You can install some debugging tools and redirect the logs to the resultdirectory.

Eg: You can use AddResultFile method of TestContext for adding results file to the blob as well:

       [TestMethod]

        public void TestMethod3()

        {

            TestContext.AddResultFile(filePath);

        }

   You can use the following code to copy files directly to result directory :

       [TestMethod]

        public void TestMethod4()

        {

            var sourceLocation = "C:\location";

            var resultLocation = Environment.GetEnvironmentVariable("ResultsDirectory", EnvironmentVariableTarget.Process);

            var sourceFilePath = Path.Combine(sourceLocation, "File.txt");

            var destinationFilePath = Path.Combine(resultLocation, "File.txt");

            File.Copy(sourceLocation, destinationFilePath);           

        }

How can you access the collected files from blob? 

We have exposed the connection string of the blob location which contains diagnostic files using our Test Results API.

Link to our Test Results API Documentation: http://www.visualstudio.com/en-us/integrate/reference/reference-vso-clt-overview-vsi

Eg:

GET https://{account}.vsclt.visualstudio.com/_apis/clt/testruns/{testrunid}/results

Response:

{

    "resultsUrl":"https://{serviceStore}.blob.core.windows.net/ets-containerfor-db236546-06d8-406c-8c76-10296ab9ef56/f471ec15-95e2-42f7-8f97-92b763581e02/TestResult/LoadTest.ltrar.zip?sv=2014-02-14&sr=b&si=sas_tenant_policydb236546-06d8-406c-8c76-     10296ab9ef56&sig=TFPOQOCQ8uNu2%2FYFrW4F%2BTeXiiPiAIlZPCI6jyi%2F5Dw%3D&se=2015-01-29T04%3A43%3A58Z",

    "diagnostics":

    {

        "diagnosticStoreConnectionString":"https://{serviceStore}.blob.core.windows.net/ets-containerfor-db236546-06d8-406c-8c76-10296ab9ef56?sv=2014-02-14&sr=c&sig=LX1lgAUYKXbN%2BphGx0a%2B5tfQM6PWBUtFch%2BZnWQOtmo%3D&se=2015-01-29T04%3A43%3A59Z&sp=rl",

        "relativePathToDiagnosticFiles":"f471ec15-95e2-42f7-8f97-92b763581e02/TestResult/Diagnostics"

    },

    "counterGroups":[{"groupName":"Performance","url":"https://{Account}.vsclt.visualstudio.com/_apis/clt/testRuns/f471ec15-95e2-42f7-8f97-92b763581e02/CounterInstances?groupNames=Performance"},{"groupName":"Throughput","url":"https://{Account}.vsclt.visualstudio.com/_apis/clt/testRuns/f471ec15-95e2-42f7-8f97-92b763581e02/CounterInstances?groupNames=Throughput"},{"groupName":"Application","url":"https://{Account}.vsclt.visualstudio.com/_apis/clt/testRuns/f471ec15-95e2-42f7-8f97-92b763581e02/CounterInstances?groupNames=Application"}]

}

The highlighted section are details required to download the diagnostics data.

Link to Azure Blob documentation: http://azure.microsoft.com/en-in/documentation/articles/storage-dotnet-how-to-use-blobs/

Using the above information, you can use below code for downloading the files from blob:

 public void DownloadFilesFromBlob()

 { 

            var container = new CloudBlobContainer(diagnosticStoreConnectionString);

            Microsoft.WindowsAzure.StorageClient.CloudBlobDirectory blobDirectory = container.GetDirectoryReference(relativePathToDiagnosticFiles);

            var blobFiles = blobDirectory.ListBlobs(new BlobRequestOptions { UseFlatBlobListing = true }).ToList();

            string dirPath = Path.GetTempPath();

            dirPath = dirPath + "/diagnostics";

            Directory.CreateDirectory(dirPath);

            foreach (var listBlobItem in blobFiles)

            {

                var blob = (CloudBlob)listBlobItem;                               

                var path = dirPath + "/" + Path.GetFileName(blob.Name);

                blob.DownloadToFile(path);

           }

}

Once the files are downloaded to your machine, you can analyze the files.

Key features of the diagnostic feature

• Maximum of 10 MB size per agent data will be collected from result directory.
• The contents of the resultdirectory are copied at regular intervals except for trace logs which are uploaded at the end of run.
•  

Please contact team if you need the max limit per agent to be increased at vsoloadtest@microsoft.com

Hope the post helps in debugging your tests on CLT.