Azure Data Factory Updates: Execute ADF Custom .NET activities using Azure Batch

We are continuously listening to your feedback and adding new improvements to Azure Data Factory service. Earlier today, we released a new feature which allows customers to run Custom .NET activities using Azure Batch as a compute resource.

Customers will have to create their own Azure Batch pools and specify the number of VM’s along with other configurations. Azure Batch pools provides the following features to customers:

1. Create pools containing a single core to thousands of cores.
2. Auto scale VM count based on a formula
3. Support VMs of any size
4. Configurable number of tasks per VM
5. Queue unlimited number of tasks

In order to use Azure Batch to run the ADF Custom .Net Activity, follow the steps below:

• Create your own Azure Batch pool:
  • Create an Azure Batch Account by following the steps here.
  • Once the Azure Batch Account is created, fetch the Batch Account key by clicking ‘Batch Services’ in Azure portal (same process as you fetch keys for a storage account).
  • Create your own Azure Batch pool. In order to perform this step, download the Azure Batch Explorer from this location. Open the C# solution and build it. Launch the ‘BatchExplorer.exe’
  • Add your ‘Azure Batch Account’ created above using the Batch explorer interface.

Figure 1: Add ‘Azure Batch Account’

• Connect to your Azure Batch account.

Figure 2: Connect Azure Batch Account

• Create your Batch pool by specifying the number of VMs along with other configurations. Tools-> Create Pool. Note: You can also create pools using Azure Batch library for .Net.

Figure 3: Create Azure Batch Pool

• Create ADF ‘Azure Batch Linked Service’ using the ‘Batch Account Name’, ‘Access Key’, ‘Batch Pool Name’ created in the previous step.

{
    "name": "MyAzureBatchLinkedService",
    "properties":
    {
        "type": "AzureBatchLinkedService",“accountName”: “”,
        “accessKey”: “”,
        “poolName”: “”,
        “linkedServiceName”: “”
    }
}

Figure 4: ADF- Azure Batch Linked Service Sample

• Associate this linked service with the custom activity. You do this by specifying the linked service as the value for the linkedServiceName property in the activity definition of pipeline JSON. See ‘Use custom activities in an Azure Data Factory pipeline’ for more details.

Figure 5: Use Azure Batch Linked Service in ADF Custom .Net Activity

• Once the Custom .Net Activity runs using Azure Batch Compute in Azure Data Factory, use the Azure Portal or Azure PowerShell to monitor your datasets.

Figure 6: ADF Monitoring View for ‘OutputTableForCustom’ dataset produced using Azure Batch Compute

By adding Azure Batch as a compute resource, we are now supporting both Azure Batch and HDInsight for running Custom .NET activities in Azure Data Factory. If you are missing a specific functionality or encounter any issues, please visit the Azure Data Factory Forums and provide your feedback.