When you use Azure Automation to create and run runbooks, two key tasks are the testing of runbooks during authoring and the troubleshooting of runbook jobs in production. In both cases, you need your runbooks to produce useful log messages so that you can learn what is going on during execution. Fortunately, PowerShell Workflow, the engine that powers Azure Automation runbooks, has multiple message streams that you can control for optimal benefit during testing and troubleshooting. In this post, I will cover some concepts and best practices for controlling runbook streams that will help you debug and troubleshoot your runbooks.
Testing in Azure Automation
The Azure Automation runbook authoring page is where you create runbooks. It has a pane for editing PowerShell Workflow code, a pane for viewing test output, and controls for inserting activities, runbooks, and assets (see the screenshot below in the Runbook Output for Debugging section). As you author a runbook you will probably test it at key points to assure that the functionality works as expected. Initiating a test for a draft runbook is easy: you simply click the Test button. When the test job is initiated, the Output Pane appears and streams from the runbook are written into this pane. Of course, the information you put into the streams and the particular streams you use are critical for how well you can debug the runbook. To help you with this, in the next sections I will show you how to control the various streams that PowerShell Workflow makes available.
Controlling Runbook Streams
PowerShell Workflow supports writing records to six different streams – each with a different purpose. Runbooks can write to these streams using these cmdlets:
- Write-Output
- The output objects that result from the runbook processing
- These are often meant as return data to be used by parent runbooks that called this runbook.
- Write-Warning
- Warning messages
- Write-Error
- Error messages
- Errors are different from exceptions – by default errors will not stop runbook execution, whereas uncaught exceptions will cause the runbook to suspend.
- Write-Verbose
- Verbose records for general messages and debugging info
- Write-Progress
- Progress records are automatically generated by Workflow at the start and end of each activity in a runbook.
- Write-Debug
- This is meant for PowerShell console use, so do not use in Azure Automation.
By learning when to use each stream, you can maximize their usefulness to you during the authoring/debugging phase of runbook creation and during the troubleshooting of production runbook jobs.
In Azure Automation there are two main places where you can view runbook streams – the runbook authoring page and the runbook job page (see screenshots below). In the runbook authoring page you will want to view the streams as you debug the runbook during creation and testing. In the runbook job page you will want to view streams when troubleshooting runbook job issues after the runbook has been published and is running in production.
Runbook Output for Debugging
During runbook authoring, when you test a runbook, by default only the Output, Warning, and Error streams are written to the Output Pane. In the screenshot below you can see this behavior:
The reason that only these three streams are written, is related to the default preferences set for the streams (see this article on PowerShell preference variables for more information). As you can see in the screenshot below, the default preferences for Warning and Error are “Continue”, while the preference for Verbose is “SilentlyContinue”. This means that when the Write-Verbose cmdlet is used the message will not be written to the Output Pane (unless the cmdlet call contains the –Verbose switch).
Changing the default stream settings for testing
During testing you can override the default preferences for the Verbose stream.
- To view all Verbose messages in the Output Pane you can set $VerbosePreference=”Continue” at the top of the runbook.
- If you only want to see select Verbose messages you can add the “-Verbose” switch to specific Write-Verbose calls (or to any cmdlet with Verbose messages).
The screenshots below shows this preference being set in a runbook and the results.
Best Practice: Use Write-Verbose for Azure Automation runbook debugging during authoring. Don’t use Write-Debug or Write-Output for debug messages: Write-Debug is for interactive console use only, and Write-Output is meant only for output objects that will be used as input to another process (such as a runbook that invoked this runbook inline as a child runbook).
Best Practice: Use Write-Verbose for general comments you want to appear in the job output for testing and troubleshooting. Again, never use Write-Output for this.
Best Practice: Don’t use Write-Progress in your runbooks. Progress records never appear in the test Output Pane (because they are meant to be displayed in a console progress bar), thus they are not useful for debugging. In the next section about production jobs, you will see that there may be some use for Progress records when troubleshooting, but only the Progress records automatically generated by the PowerShell Workflow engine.
Runbook Output for Production Job Logs
After you have finished authoring your runbook and have published it for use in production, you should think about the amount of information you need to troubleshoot any possible runbook job issues.
For production jobs (just like test jobs) by default only the Output, Warning, and Error records are written to the database for each job. The reason these records are stored by default is that they are the most important pieces of information needed on a routine basis. Note that if you set the Warning or Error preference variables to ”SilentlyContinue” within a runbook and publish it, then in production execution the runbook will not output any messages for these streams, even if the runbook configuration has logging enabled for these streams.
Below are two screenshots for the Dashboard and History pages of a job which illustrate how the Output, Warning, and Error records appear by default. The Progress and Verbose records are not stored by default in order to save storage space. (For more information see the blog on Monitoring and Troubleshooting Your Runbooks.)
Changing the default stream settings for production troubleshooting
You can change the default settings for the logging of Verbose and Progress records in the Configure page for each runbook. See the screenshot below where these settings have been changed to enable logging.
Note that these configuration settings apply only during production runs of the published runbook, and are not obeyed in the runbook authoring/testing page while the runbook is in draft mode. You can also change these settings using the Set-AzureAutomationRunbook cmdlet.
Best Practice: Use Write-Verbose for general, progress, or debugging information that you want to appear in the job history for troubleshooting production Azure Automation runbooks. Don’t use Write-Debug or Write-Output for this information.
In the screenshot below you can see the result of enabling the logging of Verbose and Progress records.
You can see that much more information is now available in the Job History page. This extra information can be essential for troubleshooting production problems with a runbook.
Best Practice: Keep the default setting of Off for the Verbose and Progress streams, and turn them on only when you need to troubleshoot issues.
The Progress records are especially numerous. By default the workflow engine writes two Progress records for each activity in the workflow – one Progress record when the activity starts and one Progress record when the activity completes. Unless you need this information to track the progress of a runbook for troubleshooting, you should make sure that Progress logging is turned off.
Best Practice: Don’t use Write-Progress in your runbooks. The Progress records that are automatically generated by the PowerShell workflow engine provide plenty of information if you need to track progress during troubleshooting.
Summary
As you can see, it is easy to control the logging streams for runbooks to enable optimal debugging and troubleshooting. Stick to the best practices and you should have problem-free control over debugging and troubleshooting your runbooks.
Not an Azure Automation user yet? Sign up for the Preview and then check out the Getting Started Guide.