Announcing the 1.0.0 RTM of Microsoft Azure WebJobs SDK

Download in this Release

The SDK is pre-installed in the Azure WebJob project template which shipped with Visual Studio 2013 Update 4 RC

You can download the WebJobs SDK from the NuGet gallery and can install these packages from the NuGet gallery using the NuGet Package Manager Console, like this:

Install-Package Microsoft.Azure.WebJobs

If you want to use Microsoft Azure Service Bus triggers, install the following package:

Install-Package Microsoft.Azure.WebJobs.ServiceBus

What is the WebJobs SDK?

The WebJobs feature of Microsoft Azure Web Sites provides an easy way for you to run programs such as services or background tasks in a Web Site. You can upload and run an executable file such as an .exe, .cmd, or .bat file to your web site while running these as triggered or continuous WebJobs. Without the WebJobs SDK, connecting and running background task requires a lot of complex programming. The SDK provides a framework that lets you write a minimum amount of code to get common tasks done.

The WebJobs SDK has a binding and trigger system which works with Microsoft Azure Storage Blobs, Queues and Tables as well as Service Bus. The binding system makes it easy to write code that reads or writes Microsoft Azure Storage objects. The trigger system calls a function in your code whenever any new data is received in a queue or blob.

Scenarios for the WebJobs SDK

Here are some typical scenarios you can handle more easily with the Azure WebJobs SDK:

Image processing or other CPU-intensive work.
Other long-running tasks that you want to run in a background thread, such as sending emails. Until now you couldn’t do this in ASP.NET because IIS would recycle your app if your app was idle for some time. Now with Always On in Azure Web Sites you can keep the web site from being recycled when the app is idle. Always On ensures that the site does not go to sleep, which means you can run long-running tasks or services using WebJobs and the WebJobs SDK.
Queue processing. A common way for a web frontend to communicate with a backend service is to use queues. This is a common producer – consumer pattern.
RSS aggregation. If you have a site that maintains a list of RSS feeds, you could pull in all of the articles from the feeds in a background process.
File maintenance, such as aggregating or cleaning up log files.
Ingress, such as CSV readers, parsing logs and storing data in Tables.

Goals of the SDK

Provide a way to make it easier to use Azure Storage when doing any background processing work.
The SDK makes it easier to work with Azure Storage. You do not have to deal with writing code to read/ write from storage.
Provide a rich diagnostics and monitoring experience without your having to write any diagnostics and logging code.

Features of the SDK

Triggers

Functions will be executed when a new input is detected on a queue or a blob.

Bindings

The SDK supports binding to provides model binding between C# primitive types and Azure storage like Blobs, Tables, Queues and Service Bus. This makes it easy for a developer to read/ write from blobs, tables and queues. This provides the following benefits.

Convenience. You can pick the type that’s most useful for you to consume and the WebJobs SDK will take care of the glue code. If you’re doing string operations on a blob, you can bind directly to TextReader/TextWriter, rather than worry about how to convert to a TextWriter.
Flushing and Closing: The WebJobs SDK will automatically flush and close outstanding outputs.
Unit testability. The SDK makes it possible to unit test your code since you can mock primitive types like TextWriter rather than ICloudBlob.
Diagnostics. Model binding works with the dashboard to give you real time diagnostics on your parameter usage.

The following Bindings are currently supported: Stream, TextReader/ Writer, and String. You can add support for binding to your custom types and other types from the Storage SDK as well.

Azure Queues

The SDK can be used to trigger a function on a new message on a queue. The SDK allows you to easily access the message contents by allowing you to bind to String, Poco (Plain old CLR object), byte[] and Azure storage SDK types. Following are the other mainline features available for queues.

Please read these announcement posts 0.5.0-beta, 0.4.0-beta and 0.3.0-beta for more details.

Trigger a function and bind the message contents to String, Poco (Plain old CLR object), byte[] and CloudQueueMessage.
Send single or multiple message on queue.
Parallel execution with Queues: The SDK will fetch messages for a queue in parallel within a QueueTrigger. This means if a function is listening on a queue as shown below, the SDK will get a batch of 16 (default) queue messages in parallel for this queue. The function is also executed in parallel
Handling of poison messages in Azure Queues
Access DequeueCount property of the queue
Better polling logic for Azure Queues: The SDK implements a random exponential back-off algorithm to reduce the effect of idle queue polling on storage transaction costs.
Fast path notifications: The SDK will fast track messages if you are sending messages to multiple queues using the SDK.
Configuration options for queue polling: The SDK exposes a few knobs where you can configure the Queue polling behavior.
- MaxPollingInterval used for when a queue remains empty, the longest period of time to wait before checking for a message to. Default is 1 min.
- MaxDequeueCount used for when the Queue message is moved to a poison queue. Default is 5,

Azure Blobs

The SDK can be used to trigger a function when a new blob is detected or an existing blob is updated. The SDK allows you to access the blob contents by allowing you to bind to Stream, String, Poco (Plain old CLR object), byte[], TextReader, TextWriter and Azure storage SDK types.

Please read these announcement posts 0.5.0-beta, 0.4.0-beta and 0.3.0-beta for more details.

BlobTriggers are only triggered when a new blob is detected or an existing blob is updated.
Retry and Error handling for Blobs: This release of the SDK adds support for retrying functions when there was an error processing a blob. A BlobTrigger will be processed up to a specified maximum number of retries (the default is 5 times). When the threshold value is reached and the function is executed 5 times, the SDK will put a message in the queue called “webjobs-blobtrigger-poison”. You can trigger a function using QueueTrigger on this queue and do your custom error handling of the message.

Azure Storage Tables

The SDK allows you to bind to Tables and perform read, write, update and delete operations.

Please read these announcement post for 0.6.0-beta, 0.5.0-beta, 0.4.0-beta and 0.3.0-beta for more details.

Ingress is a common scenario when you are parsing files stored in blobs and storing the values in Tables such as CSV readers. In these cases the Ingress function could be writing lots of rows (million in some cases).

The WebJobs SDK makes it possible to implement this functionality easily and allows add real time monitoring capabilities such as number of rows written in the table so you can monitor the progress of the Ingress function.

Azure Service Bus

Similar to Azure Queues the SDK allows you to trigger functions when a new message is sent to Service Bus Queue or topic. The SDK allows you to easily access the message contents by allowing you to bind to String, Poco (Plain old CLR object), byte[] and BrokeredMessage.

Please read this announcement post for 0.3.0-beta for more details.

General

Following are some of the other features of the SDK.

Async Support: The SDK supports async functions.
CancellationTokens: The functions can take a CancellationToken parameter in your function and the function can receive a cancellation request from the Host
NameResolver: The SDK has an extensibility layer which allows you to specify the source for queue names or blob names. For example you can use this feature to pick up queue names from a config file. Look at this sample.
WebJobs Shutdown notification: WebJobs has a graceful shutdown notification feature which raises a notification when the WebJob is stopping. The SDK supports this graceful shutdown in WebJobs by notifying you when the WebJob is shutting down. This information is flowed to the function using the CancellationToken. The following function takes in a CancellationToken which will receive the Cancellation request when the WebJob is stopping.

Dashboard for monitoring WebJobs

As WebJobs execute, you can monitor them in real time. You can see their state (Running, Stopped, Successfully completed), last run time and the logs of a particular execution. The following screenshot shows you a view of all WebJobs running in your Website.

When you write a WebJob using the SDK, you get diagnostics and monitoring experience for the functions in your program. For example, let’s say that you have an Image processing WebJob called “ImageResizeAndWaterMark” that has the following flow.

When a user uploads an image to a Blob container called “images-input”, the SDK will trigger Resize function. Resize will process the image and write to “images2-output” container which will trigger the Watermark function. Watermark function will resize the image and write it to “images3-output” Blob container. The following code shows the WebJob described above.

public class ImageProcessing
    {
        public static void Resize(
            [BlobTrigger(@"images-input/{name}")] WebImage input,
            [Blob(@"images2-output/{name}")] out WebImage output)
        {
            var width = 80;
            var height = 80;
            output = input.Resize(width, height);
        }

        public static void WaterMark(
            [BlobTrigger(@"images2-output/{name}")] WebImage input,
            [Blob(@"image3-output/{name}")] out WebImage output)
        {
            output = input.AddTextWatermark("WebJobs", fontSize: 6);
        }
    }

    public class WebImageBinder : ICloudBlobStreamBinder
    {
        public Task ReadFromStreamAsync(Stream input, System.Threading.CancellationToken cancellationToken)
        {
            return Task.FromResult(new WebImage(input));
        }

        public Task WriteToStreamAsync(WebImage value, Stream output, System.Threading.CancellationToken cancellationToken)
        {
            var bytes = value.GetBytes();
            return output.WriteAsync(bytes, 0, bytes.Length);
        }
    }

When you run the WebJob in Azure, you can view the WebJobs Dashboard by clicking the logs link of the “ImageResizeAndWaterMark” in the WEBJOBS tab of Windows Azure Websites portal.

Since the Dashboard is a SiteExtension you can access it by going to the url: https://mysite.scm.azurewebsites.net/azurejobs. You will need your deployment credentials to access the SiteExtension. For more information on accessing Site Extension, see the documentation on the Kudu project https://github.com/projectkudu/kudu/wiki/Accessing-the-kudu-service

Function execution details

When you are monitoring a particular execution of this “ImageResizeAndWaterMark” WebJob, you can view invocation details about the functions in the program such as:

What are the parameters of this function?
How much time did it take for the function to execute.
How much time did it take to read from Blob and how many bytes were read/ written.

Invoke & Replay

In the above example if the WaterMark function fails for some reason, you can upload a new image and Replay WaterMark function, which will trigger the execution chain and call Resize function as well. This is useful to diagnose and debug an issue when you have a complicated graph for chaining functions together. You can also Invoke a function from the dashboard.

Causality of functions

In the above example, we know that when the WaterMark function writes to a Blob, it will trigger the Resize function. The dashboard will show this causality between functions. If you have chained lots of functions which will get triggered as new inputs are detected then it can be useful to see this causality graph.

Search Blobs

You can click on Search for a Blob and get information on what happened to that Blob. For example, in the case of the ImageResizeAndWaterMark, the Blob was written because the WaterMark function got executed. For more details on Search Blobs see this post.

Samples

Samples for WebJobs SDK can be found at https://github.com/Azure/azure-webjobs-sdk-samples

You can find samples on how to use triggers and bindings for blobs, tables, queues and Service Bus.
PhluffyShuffy which is an Image processing Website where a customer can upload pictures which will trigger a function to process those pictures from blob storage.
PhluffyLogs is an example where a WebJob parses log files generated by the app and archives the log files.

Documentation

Deploying WebJobs with SDK to Azure Websites

Visual Studio 2013 Update 3 with Azure SDK 2.4 added Visual Studio Tooling support to publish WebJobs to Azure Websites. For more information, see How to Deploy Azure WebJobs to Azure Websites

Give feedback and get help

If you have questions, you can ask them on the Azure forum, the ASP.NET forum, or StackOverflow.com. Use #AzureWebJobs SDK for Twitter and the tag Azure-WebJobsSDK for StackOverflow.