We are happy to announce the release of the Visual Studio extension for Azure Data Factory (ADF). With this release, customers can author and deploy data pipelines using the rich Visual Studio interface.
Our goal is to simplify the ADF authoring experience and remove on-boarding and deployment challenges. The ADF extension in Visual Studio improves productivity and efficiency for both new and advanced users with tailored experiences and rich tooling.
ADF Visual Studio extension is integrated with the Azure SDK 2.7 and above for Visual Studio 2013.
Features enabled in this release:
- Template based authoring: Select use-cased based templates, data movement templates or data processing templates to deploy an end-to-end data integration solution and get started hands-on quickly with Data Factory
- Integration with Solution Explorer for authoring and deploying Data Factory entities: Create & deploy pipelines and related entities as Visual Studio projects
- Integration with Diagram view for visual interaction while authoring: Visually author pipelines and datasets with aid from the Diagram view
- Integration with Server explorer for browsing and interaction with already deployed entities: Leverage the Server Explorer to browse already deployed data factories and corresponding entities. Import a deployed data factory or any entity (Pipeline, Linked Service, Datasets) into your project
- JSON editing with schema validation and rich intellisense: Efficiently configure and edit JSON documents of Data Factory entities with rich intellisense and schema validation
- Multi-Environment publishing: Publish authored pipelines to Dev, Test or Prod environment by creating separate config files for each environment
- Integrated support for deploying transformation activity execution code: Hive scripts, Pig scripts, Map Reduce & Hadoop Streaming code can be included in the Data Factory project and deployed automatically as part of the publishing experience. Support for referencing C# Project for publishing and managing .Net Activity
- Migrate data factory across multiple subscriptions: Efficiently migrate an existing data factory to a new subscription by using the export capability in the Server Explorer
Pre-requisites
You must have the following installed on your computer:
- Install Visual Studio 2013 (Update 4)
- Install Windows Azure SDK for .NET – 2.7
Note
We are continuously releasing new updates to the ADF Visual Studio extension. Once you install the Windows Azure SDK, please check for the Data Factory updates in the Visual Studio by following the instructions
- Click Tools on the menu and select Extensions and Updates
- Select Updates in the left pane and then select Visual Studio Gallery
- Select Azure Data Factory tools for Visual Studio and click Update. If you do not see this entry, you already have the latest version of the tools
Walkthrough
In this walkthrough, let’s use the ADF Visual Studio extension to create a pipeline that copies data from Azure SQL to Azure Blob Storage.
- Launch Visual Studio 2013. Click File, point to New, and click Project. You should see the New Project dialog box.
- In the New Project dialog, select the DataFactory template, and click Data Factory Templates.
Figure 1: Data Factory project templates
- Displays the list of the Data Factory Templates. You can see end to end use case templates, data movement templates and data processing templates. Let’s select the Azure SQL to Azure Blob template and leave the Include sample data option checked. This would ensure we can deploy this pipeline with sample data.
Figure 2 : Data Factory solution template gallery
Note
If you are not already signed-in, please sign in with an account that is associated with Azure subscription
- Create a new data factory. You can also select an existing data factory in your subscription. Leave Publish to Data Factory option selected which will automatically deploy the pipeline to the data factory
Figure 3: Data Factory Configuration – create or select data factory
- Configure the Azure SQL source and the Azure Blob Storage destination resources for the pipeline. You can either select an existing resource or create new resource. In this example, let’s create new resources
Figure 4: Data Factory Configuration – configure Azure resources
- Review the configuration settings
Figure 5: Data Factory Configuration – configuration summary
- Clicking Next provisions all the Azure resources, creates the data factory pipeline and all related entities (Linked Services and Datasets) and deploys the pipeline
Figure 6: Data Factory resources deployment
- Once the deployment steps are completed, click Finish to close the wizard and land on the Visual Studio canvas. At this stage, the pipeline is created in the data factory and running. Navigate to the Azure portal for monitoring the pipeline
- Continuing with the Visual Studio flow, the canvas displays the pipeline diagram and the corresponding JSON code. The Solution Explorer includes the data factory project for this pipeline. You can view the Linked Services and the Datasets by clicking the JSON files and opening them on the canvas
Figure 7: Data Factory Visual Studio extension canvas
- You can edit the pipeline by using the JSON editor with rich intellisense and deploy the changes to the factory by clicking the Publish button on the toolbar
Figure 8: Data Factory Visual Studio extension publish experience
- Use the Server Explorer to view deployed data factories in your subscription. Export a data factory or any data factory entity locally as a new Visual Studio project or to an existing project
Figure 9: Data Factory Visual Studio extension Server Explorer experience
For additional functionalities of the ADF Visual Studio extension including starting with an Empty Data Factory project, visit the product documentation located here.
We are looking forward to hear your feedback on the ADF Visual Studio extension. If you are missing a specific functionality or encounter any issues, please visit the Azure Data Factory Forums and provide your feedback.