The availability of the OMS Search API has enabled automation scenarios which involves alerting, notification and potential remediation based on log analytics data in Operations Management Suite (OMS).
In this blog post we will talk about a sample scenario for monitoring a honeypot account with an Azure Automation runbook . This runbook checks if someone has attempted to logon with this account on a specific server. A honeypot account is often used to entice malicious attackers to use a specific account for which monitoring is setup. In addition we will also look into using a Hybrid Runbook Worker for not only leveraging my on-premises resources, but also to overcome potential challenges when there is no PowerShell module to upload to Azure Automation.
Our runbook will talk to the OMS Search API and is able to execute searches or can retrieve the results of a saved search. Based on the query results we can decide to take corrective actions or to invoke other runbooks for alerting and notification purposes. During conversations with customers we have been asked many times if OMS has similar System Center Operations Manager (SCOM) task integration to take actions based on OMS search query results. This blog post enables you to explore those options. One thing to point out though is that when using SCOM, the agent locally checks for a certain condition to be true and acts upon that, while in the OMS scenario the information gets sent to OMS first, is indexed and analyzed so that we can query against that data.
As mentioned in the OMS Search API announcement you have three different options to leverage the API:
In this blog post I’m going to use the ARMClient which you can download from Github. The ARMClient cannot really be used as a PowerShell module “as is” and uploaded to Azure Automation. This however is a great opportunity to test the Azure Automation Hybrid Runbook Worker which can overcome this challenge because the runbook is executed locally and has therefore access to local resources like our ARMClient.
Ok, time to set up the basics
So what do we need to make this happen?
The basics:
- First we obviously need to have an OMS subscription. If you haven’t got one, sign up here for a free trial!
- We need to have an OMS agent reporting to OMS to provide search data
- Create, test and save your custom query in OMS and make sure you get data returned
- Lastly we need an Azure Automation account for our OMS Search API runbook
In this blog post we will walk through the following:
- Setup a Hybrid Runbook Worker
- Install the ARM Client
- Author an OMS Search API runbook
- Create a saved search in OMS
- Monitoring our Honeypot Account with an Azure Automation runbook
1. Hybrid Runbook Worker Setup
Brian Wren wrote an excellent post how to setup and configure a Hybrid Runbook Worker. As described you can leverage any OMS agent to become a Hybrid Runbook Worker. Just login to one of your OMS agents, start an administrator PowerShell session and type the following:
Import-Module HybridRegistration Import-Module "C:Program FilesMicrosoft Monitoring AgentAgentAzureAutomationFilesHybridRegistrationHybridRegistration.psd1" Add-HybridRunbookWorker –Name-EndPoint -Token
Shortly your Hybrid Runbook Worker will show up in the Azure Automation portal:
2. The ARMClient installation
For the ARMClient installation you first need to install Chocolatey and then the ARMClient package which is described here.
Make sure that before you run the Chocolatey installation you update your System Variables to control where Chocolatey will be installed so that the Azure Automation runbook is able to find and access the ARMClient.
Since we are going to use the ARMClient in an Automation runbook we are going to use a Service Principal which is more automation friendly since we do not have to logon. The setup and configuration is described here.
When this has been setup I would recommend to first test your ARMClient PowerShell script locally on the Hybrid Runbook Worker to ensure that the ARMClient successfully can connect to OMS.
For testing your PowerShell script using the ARMClient you need to have the following information:
- Service Principal info: tenantID, username and password
- Your Azure subscription ID
- The OMS workspace name
- The OMS resource group ID
After a successful installation of the ARMClient you can run the following PowerShell script on the Hybrid Runbook Worker to test connectivity:
$AzureSubscription = "YourAzureSubscription" $OMSResourceGroupId = "YourOMSResourceGroupID" $OMSWorkspaceName = "YourOMSWorkspaceName" #login with Service Principal, syntax armclient, spn(=TenantID), Username, Password $SPNforOMS = "YourSPN" $AzureUserNameForOMS = "YourAzureUserNameForOMS" $AzureUserPasswordForOMS = "YourAzureUserNameForOMSPassword" armclient spn $SPNforOMS $AzureUserNameForOMS $AzureUserPasswordForOMS armclient get /subscriptions/$AzureSubscription/resourceGroups/$OMSResourceGroupId/providers/Microsoft.OperationalInsights/workspaces?api-version=2015-03-20
When your run this PowerShell script you will get something like this:
3. Author an OMS Search API runbook
Now that we have setup the ARMClient and tested connectivity to our OMS workspace on our Hybrid Runbook Worker its time to author our first OMS Search API runbook.
You need to create a number of assets in Azure Automation which are actually similar to the previous step where you’ve tested the ARMClient locally:
Asset Type Name Description Credential AzureUserNameForOMS Your OMS username and password Variable AzureSubscription Your Azure subscription ID Variable OMSResourceGroupID Your OMS Resource Group ID Variable OMSWorkspaceName Your OMS workspace name Variable SPNforOMS Your Service Principal Name for OMS
For our first OMS Search API runbook I’m going to leverage a dynamic search query which I can pass as a parameter ($query) and also start/end dates with time ranges to limit our results:
workflow Invoke-OMSsearchQuery
{
param
(
[Parameter(Mandatory=$false)]
[string]$StartDate = "2015-07-10",
[Parameter(Mandatory=$false)]
[string]$StartTime = "09:00",
[Parameter(Mandatory=$false)]
[string]$EndDate = "2015-07-15",
[Parameter(Mandatory=$false)]
[string]$EndTime = "09:00",
[Parameter(Mandatory=$false)]
[string]$query = "Type=SecurityEvent EventID=4625 | Measure Count() by TargetAccount"
)
#Retrieve stored variables
$AzureSubscription = Get-AutomationVariable -Name 'AzureSubscription'
$OMSWorkspaceName = Get-AutomationVariable -Name 'OMSWorkspaceName'
$OMSResourceGroupId = Get-AutomationVariable -Name 'OMSResourceGroupId'
Write-Output "Executing runbook on hybrid runbook worker: $env:ComputerName"
#login with a Service Principal, syntax armclient, spn(=TenantID), Username, Password
$SPNforOMS = Get-AutomationVariable -Name 'SPNforOMS'
$cred = Get-AutomationPSCredential -Name 'AzureUserNameForOMS'
$null = ARMClient.exe spn $SPNforOMS $cred
#Use a dynamic OMS search
$QueryRange = $StartDate + "T" + $StartTime + ":00.231Z','end':'" + $EndDate + "T" + $EndTime + ":00.231Z'"
$query = "{'query':'$query','start':'$QueryRange}"
$json = armclient post /subscriptions/$AzureSubscription/resourceGroups/$OMSResourceGroupId/providers/Microsoft.OperationalInsights/workspaces/$OMSWorkspaceName/search?api-version=2015-03-20 $query | ConvertFrom-Json
$queryResults = $json.value
Write-Output "**** Executing query*** " $query
Write-Output "Query results: " $queryResults
}Let’s publish the runbook and run it on our Hybrid Runbook Worker. We will use the following query which will return computers which have rebooted or have been shutdown:
Type: Event shutdown EventLog:System NOT (Source:Microsoft-Windows-WAS) Source: User32 EventID: 1074 | Measure count () by Computer
We nicely get our query results back:
4. Create a saved search in OMS
Our honeypot account runbook will leverage a saved OMS search. This will not only limit the scope and have a precise defined search, but will also avoid syntax errors.
To save a search in OMS simply use the Log Search functionality to execute and test your search. Notice that I’m narrowing down the search to 1 computer only “WHDVM1”:
Select Save and provide a Name and Category (we will use this later in our final runbook):
5. Monitoring our Honeypot Account with Azure Automation
Now that we’ve setup and tested all the prerequisites its time to author our runbook which uses our saved OMS query to monitor and to notify us when our honeypot account has been used to attempt to logon to a specific server (like a domain controller). Since we have narrowed the saved search down to a specific server in which we are interested, we only need to build some simple logic in our runbook to filter the account. Obviously we could have leveraged a full and locked down saved query in OMS as well. For our notification part we are leveraging the Send-MailMessage PowerShell cmdlet as stored in our Send-AlertNotification runbook:
workflow Send-AlertNotification
{
param (
[Parameter(Mandatory=$true)] [string]$BodyText,
[Parameter(Mandatory=$true)] [string]$Subject
)
$Cred = Get-AutomationPSCredential -Name 'EmailCredentials'
Send-MailMessage `
-From "OMS Alerts " `
-Subject $Subject `
-To "XWING Alerts " `
-Body $BodyText `
-SmtpServer "smtp.live.com" `
-Credential $Cred `
-UseSsl
}
The Send-AlertNotification runbook uses a stored credential for our email account:
| Asset Type | Name | Description |
| Credential | EmailCredentials | Your email name and password |
In our last Azure Automation runbook Check-OMSHoneyPotLogon all the pieces are coming together. Notice that below we have our OMS saved search ($SavedSearchName), our $BodyText variable constructs our email body and finally we invoke our Send-AlertNotification runbook:
workflow Check-OMSHoneyPotLogon
{
param
(
[Parameter(Mandatory=$false)]
[string]$StartDate = "2015-07-16",
[Parameter(Mandatory=$false)]
[string]$StartTime = "09:00",
[Parameter(Mandatory=$false)]
[string]$EndDate = "2015-07-17",
[Parameter(Mandatory=$false)]
[string]$EndTime = "17:00",
[Parameter(Mandatory=$false)]
[string]$SavedSearchName = "honeypot|honeypot logon",
[Parameter(Mandatory=$false)]
[boolean]$Alert = $false
)
#Retrieve stored variables
$AzureSubscription = Get-AutomationVariable -Name 'AzureSubscription'
$OMSWorkspaceName = Get-AutomationVariable -Name 'OMSWorkspaceName'
$OMSResourceGroupId = Get-AutomationVariable -Name 'OMSResourceGroupId'
Write-Output "Executing runbook on hybrid runbook worker: $env:ComputerName"
#login with a Service Principal, syntax armclient, spn(=TenantID), Username, Password
$SPNforOMS = Get-AutomationVariable -Name 'SPNforOMS'
$cred = Get-AutomationPSCredential -Name 'AzureUserNameForOMS'
$null = ARMClient.exe spn $SPNforOMS $cred
#Use a saved search
$OMSsavedSearch = armclient get /subscriptions/$AzureSubscription/resourceGroups/$OMSResourceGroupId/providers/Microsoft.OperationalInsights/workspaces/$OMSWorkspaceName/savedSearches/"$SavedSearchName"?api-version=2015-03-20
$query = $OMSsavedSearch | ConvertFrom-Json
$query = $query.properties.Query
Write-Output "Retrieved Query: $SavedSearchName"
$QueryRange = $StartDate + "T" + $StartTime + ":00.231Z','end':'" + $EndDate + "T" + $EndTime + ":00.231Z'"
$query = "{'query':'$query','start':'$QueryRange}"
$json = armclient post /subscriptions/$AzureSubscription/resourceGroups/$OMSResourceGroupId/providers/Microsoft.OperationalInsights/workspaces/$OMSWorkspaceName/search?api-version=2014-10-10 $query | ConvertFrom-Json
Write-Output "**** Executing query*** " $query
#Check our Honeypot Account
$Accounts = $json.value.targetusername
foreach ($Account in $Accounts)
{
if($Account -eq "LocalAdmin")
{
$Alert = $true
$AccountName = $Account
}
}
if($Alert -eq $true)
{
Write-Output "Raising Alert! Logon attempt found for account: $AccountName"
$BodyText = @"
Logon attempt detected for account:
$AccountName
"@
#Invoking Send-AlertNotification runbook
Write-Output "Sending email..."
Send-AlertNotification `
-Subject "Logon attempt detected for account $AccountName" `
-BodyText $BodyText
}
}Let’s start our runbook on our Hybrid Runbook Worker and watch the output:
And here is our email:
Conclusion
With the release of the OMS Search API a number of automation scenarios have become available based on OMS analytics data. The solution sample in this blog post is just a simple example how to connect to OMS and start leveraging the rich analytics data in OMS.
Happy Automating!