Announcing the 1.0.0-rc1 of Microsoft Azure WebJobs SDK

We are releasing another preview of the Microsoft Azure WebJobs SDK, which was introduced by Scott Hanselman. To read more about the previous preview, read this announcement post.

This release has the same general feature set as 0.6.0-beta and bug fixes.

Download this release

You can download the WebJobs SDK from the NuGet gallery and also install or update these packages from the NuGet gallery using the NuGet Package Manager Console, like this:

Install-Package Microsoft.Azure.WebJobs –Pre

If you want to use Microsoft Azure Service Bus triggers, install the following package:

Install-Package Microsoft.Azure.WebJobs.ServiceBus -Pre

What is the WebJobs SDK?

The WebJobs feature of Microsoft Azure Web Sites provides an easy way for you to run programs such as services or background tasks in a Web Site. You can upload and run an executable file such as an .exe, .cmd, or .bat file to your web site while running these as triggered or continuous WebJobs. Without the WebJobs SDK, connecting and running background task requires a lot of complex programming. The SDK provides a framework that lets you write a minimum amount of code to complete common tasks.

The WebJobs SDK has a binding and trigger system which works with Microsoft Azure Storage Blobs, Queues and Tables as well as Service Bus. The binding system makes it easy to write code that reads or writes Microsoft Azure Storage objects. The trigger system calls a function in your code whenever any new data is received in a queue or blob.

Scenarios for the WebJobs SDK

Here are some typical scenarios you can handle more easily with the Azure WebJobs SDK:

• Image processing or other CPU-intensive work.
• Other long-running tasks that you want to run in a background thread, such as sending emails. Until now, you couldn’t do this in ASP.NET because IIS would recycle your app if your app was idle for some time. Now with AlwaysOn in Azure Web Sites, you can keep the web site from being recycled when the app is idle. AlwaysOn ensures that the site does not go to sleep, which means you can run long-running tasks or services using WebJobs and the WebJobs SDK.
• Queue processing is a common way for a web frontend to communicate with a backend service is to use queues. This is a common producer – consumer pattern.
• RSS aggregation is great if  you have a site that maintains a list of RSS feeds, you could pull in all of the articles from the feeds in a background process.
• File maintenance, such as aggregating or cleaning up log files.
• Ingress, such as CSV readers, parsing logs and storing data in Tables.

Goals of the SDK

• Provide a way to make it easier to use Azure Storage when doing any background processing work.
• The SDK makes it easier to consume Azure Storage within your application. You do not have to deal with writing code to read/ write from storage.
• Provides a rich diagnostics and monitoring experience without having the developer write any diagnostics and logging code.

Updates in this preview

This preview has bug fixes only. The feature set is the same as before. Following is an important functional change we have made in this release.

FileAccess parameter is required when binding blobs to Stream

Starting with this release the SDK requires that the FileAccess parameter be set when using the Blob attribute and binding to a Stream.

public static void BindingToBlob(
[BlobTrigger("container/input")]  Stream input,
[Blob("container/output", FileAccess.Write)]  out Stream output
)
{

}

Change default value for MaxPollingInterval

This setting is used to configure  the longest period of time to wait before checking for a message in a queue.

The new default is 1 min instead of 10 min. We made this change so that the queue processing is responsive by default.

Service Bus triggers would not work when the message was created using string rather than stream.

If you were creating Service Bus messages in the following way, the SDK would fail to trigger a function.

var triggerMessage = new BrokeredMessage("Text");

Update min-version of JSON.NET to 6.0.4

Bind to QueueTrigger message or BlobTrigger message

The SDK allows you to bind to the queue message or blob path when a function is triggered on queues or blobs. The following code shows how you can bind to queueTrigger or blobTrigger to access the message and use it for binding.

This code shows a generic clean up function which can delete blobs once they are processed. The BlobToQueueForCleanup function has the blob path in blobTrigger (the SDK populated blobTrigger with the full path of the blob). When this function completes processing the blob path is sent on a queue called “deleteblob”.

The Cleanup function is triggered when a new message comes on “deleteblob” and the function binds to queueTrigger (the SDK populated this with the message contents) and function binds to CloudBlockBlob to delete the blob.

public static void BlobToQueueForCleanup(
    [BlobTrigger("input/{name}")] TextReader reader, string blobTrigger, 
    [Queue("deleteblob")] out string blobToDelete)
{
    blobToDelete = blobTrigger;
}

public static void Cleanup(
    [QueueTrigger("deleteblob")] string blobToDelete, string queueTrigger,
    IBinder binder)
{
    binder.Bind(new BlobAttribute(queueTrigger)).Delete();
}

Binding to blob name and extension

If you had a function where you were accessing the blob name and extension as follows, the name is everything up to the extension (including the sub directories in the path)

For eg. If the blob path was “input/foo.bar.baz.txt” then name is “foo.bar.baz” and extension is “txt”. If the blob path was “input/foo/bar/baz.txt” then name is “foo/bar/baz” and extension is “txt”.

        public static void BlobBinding(
            [BlobTrigger("input/{name}.{extension}")] TextReader input, string name,string extension,
            [Blob("output/{name}_old.{extension}",FileAccess.Write)] TextWriter writer
        )
        {
            writer.Write(input.ReadToEnd());
        }

Binding to non existent entities

Starting this release the SDK will pass in null for entities that do not exist rather than throwing. For eg. in the following code, if reader does not exist, then the SDK will pass in null for reader.

public static void BlobBinding(
[QueueTrigger("blobname")] string input, string queueTrigger,
[Blob("input/{queueTrigger}", FileAccess.Read)] Stream reader
)
{
    //reader is null
}
}

Some other examples of this behavior is as follows:

• Always Invoke with a null ref (Stream, TextReader, String)
• Pass null Stream to ICloudBlobStreamObjectBinder in case of object binding
• Pass the reference to the SDK types (CloudBlockBlob, CloudPageBlob, CloudTabe CloudQueue)
• Create if not exists – Tables, Queue and Blob container
• IQueryable returns empty Queryable

Note: The following section just outlines the feature set available as of this release.

Features of the SDK

Triggers

Functions get executed when a new input is detected on a Queue or a Blob.

Bindings

The SDK supports binding to provides model binding between C# primitive types and Azure storage like Blobs, Tables, Queues and Service Bus. This makes it easy for a developer to read/ write from blobs, tables and queues as they do not have to learn about the code around reading/ writing from Azure Storage.

• Convenience. You can pick the type that’s most useful for you to consume and the WebJobs SDK will take care of the glue code. If you’re doing string operations on a blob, you can bind directly to TextReader/TextWriter, rather than worry about how to convert to a TextWriter.
• Flushing and Closing: The WebJobs SDK will automatically flush and close outstanding outputs.
• Unit testability. The SDK makes it possible to unit test your code since you can mock primitive types like TextWriter rather than ICloudBlob.
• Diagnostics.  Model binding works with the dashboard to give you real time diagnostics on your parameter usage.

The following Bindings are currently supported: Stream, TextReader/ Writer, and String. You can add support for binding to your custom types and other types from the Storage SDK as well.

Azure Queues

The SDK can be used to trigger a function on a new message on a queue. The SDK allows you to easily access the message contents by allowing you to bind to String, Poco (Plain old CLR object), byte[] and Azure storage SDK types. Following are the other mainline features available for queues.

Please read these announcement posts 0.5.0-beta,  0.4.0-beta and 0.3.0-beta for more details.

• Trigger a function and bind the message contents to String, Poco (Plain old CLR object), byte[] and CloudQueueMessage.
• Send single or multiple message on queue.
• Parallel execution with Queues : The SDK will fetch messages for a queue in parallel within a QueueTrigger. This means if a function is listening on a queue as shown below, the SDK will get a batch of 16 (default) queue messages in parallel for this queue. The function is also executed in parallel
• Handling of poison messages in Azure Queues
• Access DequeueCount property of the queue
• Better polling logic for Azure Queues: The SDK implements a random exponential back-off algorithm to reduce the effect of idle queue polling on storage transaction costs.
• Fast path notifications: The SDK will fast track messages if you are sending messages to multiple queues using the SDK. In 0.3.0-Beta, the SDK used to poll every ~2 sec. This meant that if your app had a chain of 20 functions (a function was writing to a Queue, triggering another function which would then write to another queue and trigger another function and so on) , then it would have taken ~40 sec to process the 20 queue messages. With these changes this now takes ~8 sec.
• Configuration options for queue polling: The SDK exposes a few knobs where you can configure the Queue polling behavior.
  • MaxPollingInterval used for when a queue remains empty, the longest period of time to wait before checking for a message to. Default is 1 min.
  • MaxDequeueCount used for when the Queue message is moved to a poison queue. Default is 5,

Azure Blobs

The SDK can be used to trigger a function when a new blob is detected or an existing blob is updated. The SDK allows you to access the blob contents by allowing you to bind to Stream, String, Poco (Plain old CLR object), byte[], TextReader, TextWriter and Azure storage SDK types.

Please read these announcement posts 0.5.0-beta,  0.4.0-beta and 0.3.0-beta for more details.

• BlobTriggers are only triggered when a new blob is detected or an existing blob is updated.
• Retry and Error handling for Blobs: This release of the SDK adds support for retrying functions when there was an error processing a blob. A BlobTrigger will be processed up to a specified maximum number of retries (the default is 5 times). When the threshold value is reached and the function is executed 5 times, the SDK will put a message in the queue called “webjobs-blobtrigger-poison”. You can trigger a function using QueueTrigger on this queue and do your custom error handling of the message.

Azure Storage Tables

The SDK allows you to bind to Tables and perform read, write, update and delete operations.

Please read these announcement post for 0.6.0-beta,  0.5.0-beta,  0.4.0-beta and 0.3.0-beta for more details.

Ingress is a common scenario when you are parsing files stored in blobs and storing the values in Tables such as CSV readers. In these cases the Ingress function could be writing lots of rows (million in some cases).

The WebJobs SDK makes it possible to implement this functionality easily and allows add real time monitoring capabilities such as number of rows written in the table so you can monitor the progress of the Ingress function.

The following function show how you can write 100,000 rows into Azure Table storage.

public static class Program
{
    static void Main()
    {
        JobHost host = new JobHost();
        host.Call(typeof(Program).GetMethod("Ingress"));
    }
    [NoAutomaticTrigger]
    public static void Ingress([Table("Ingress")] ICollector tableBinding)
    {
        // Loop to simulate Ingressing lots of rows. 
        // You would replace this with your own logic
        // of reading from blob storage and write to Azure Tables.
        for (int i = 0; i < 100000; i++)
        {
            tableBinding.Add(
            new Person() 
            { PartitionKey = "Foo", RowKey = i.ToString(), Name = "Name" }
            );
        }
    }
}
public class Person
{
    public string PartitionKey { get; set; }
    public string RowKey { get; set; }
    public string Name { get; set; }
}

Azure Service Bus

Similar to Azure Queues the SDK allows you to trigger functions when a new message is sent to Service Bus Queue or topic. The SDK allows you to easily access the message contents by allowing you to bind to String, Poco (Plain old CLR object), byte[] and BrokeredMessage.

Please read this announcement post for 0.3.0-beta for more details.

General

Following are some of the other interesting features in the SDK.

- Async Support: The SDK supports async functions.

- CancellationTokens: The functions can take a CancellationToken parameter in your function and the function can receive a cancellation request from the Host

- NameResolver: The SDK has an extensibility layer which allows you to specify the source for queue names or blob names. For eg. you can use this feature to pick up queue names from a config file. Look at this sample.

- WebJobs Shutdown notification: WebJobs has a graceful shutdown notification feature which raises a notification when the WebJob is stopping. The SDK flows this notification to your functions via cancellation tokens. The SDK supports this graceful shutdown in WebJobs by notifying you when the WebJob is shutting down. This information is flowed to the function using the CancellationToken. The following function takes in a CancellationToken which will receive the Cancellation request when the WebJob is stopping.

public static void UseCancellationToken(
    [QueueTrigger("inputqueue")] string inputText,
    TextWriter log,
    CancellationToken token)
{
      // This is a long running function which can be cancelled  
      while (!token.IsCancellationRequested)
      {
          Thread.Sleep(2000);
          log.WriteLine("Not cancelled");
      }
      log.WriteLine("cancelled");
}

Dashboard for monitoring WebJobs.

As WebJobs (written in any language and of any type) execute, you can monitor them in real time. You can see their state (Running, Stopped, Successfully completed), last run time and the logs of a particular execution. The following screenshot shows you a view of all WebJobs running in your Website.

When you write a WebJob using the SDK, you get diagnostics and monitoring experience for the functions in your program. For example, let’s say that you have an Image processing WebJob called “ImageResizeAndWaterMark” that has the following flow.

When a user uploads an image to a Blob container called “images-input”, the SDK will trigger Resize function. Resize will process the image and write to “images2-output” container which will trigger the Watermark function. Watermark function will resize the image and write it to “images3-output” Blob container. The following code shows the WebJob described above.

    public class ImageProcessing
    {
        public static void Resize(
            [BlobTrigger(@"images-input/{name}")] WebImage input,
            [Blob(@"images2-output/{name}")] out WebImage output)
        {
            var width = 80;
            var height = 80;
            output = input.Resize(width, height);
        }

        public static void WaterMark(
            [BlobTrigger(@"images2-output/{name}")] WebImage input,
            [Blob(@"image3-output/{name}")] out WebImage output)
        {
            output = input.AddTextWatermark("WebJobs", fontSize: 6);
        }
    }

    public class WebImageBinder : ICloudBlobStreamBinder
    {
        public Task ReadFromStreamAsync(Stream input, System.Threading.CancellationToken cancellationToken)
        {
            return Task.FromResult(new WebImage(input));
        }

        public Task WriteToStreamAsync(WebImage value, Stream output, System.Threading.CancellationToken cancellationToken)
        {
            var bytes = value.GetBytes();
            return output.WriteAsync(bytes, 0, bytes.Length);
        }
    }

When you run the WebJob in Azure, you can view the WebJobs Dashboard by clicking the logs link of the “ImageResizeAndWaterMark” in the WEBJOBS tab of Windows Azure Websites portal.

Since the Dashboard is a SiteExtension you can access it by going to the url:  https://mysite.scm.azurewebsites.net/azurejobs.  You will need your deployment credentials to access the SiteExtension. For more information on accessing Site Extension, see the documentation on the Kudu project https://github.com/projectkudu/kudu/wiki/Accessing-the-kudu-service

Function execution details

When you are monitoring a particular execution of this “ImageResizeAndWaterMark” WebJob, you can view invocation details about the functions in the program such as:

• What are the parameters of this function?
• How much time did it take for the function to execute.
• How much time did it take to read from Blob and how many bytes were read/ written.

Invoke & Replay

In the above example if the WaterMark function fails for some reason, you can upload a new image and Replay WaterMark function, which will trigger the execution chain and call Resize function as well. This is useful to diagnose and debug an issue when you have a complicated graph for chaining functions together. You can also Invoke a function from the dashboard.

Causality of functions

In the above example, we know that when the WaterMark function writes to a Blob, it will trigger the Resize function. The dashboard will show this causality between functions. If you have chained lots of functions which will get triggered as new inputs are detected then it can be useful to see this causality graph.

Search Blobs

You can click on Search for a Blob and get information on what happened to that Blob. For example, in the case of the ImageResizeAndWaterMark, the Blob was written because the WaterMark function got executed. For more details on Search Blobs see this post.

Samples

Samples for WebJobs SDK can be found at https://github.com/Azure/azure-webjobs-sdk-samples

• You can find samples on how to use triggers and bindings for blobs, tables, queues and Service Bus.
• There is a sample called PhluffyShuffy which is an Image processing Website where a customer can upload pictures which will trigger a function to process those pictures from blob storage.

Tutorial: Getting Started with the Windows Azure WebJobs SDK

Follow this tutorial on getting started with WebJobs SDK.

Documentation

• Channel 9 video on Making Your Jobs Easier With Windows Azure WebJobs SDK
• Introduction to WebJobs and SDK by Scott Hanselman
• A web site monitor built using WebJobs and ASP.NET by Brady Gaster
• Azure WebJobs – Recommended Resources
• WebJobs video series on Azure Friday

Deploying WebJobs with SDK to Azure Websites

Visual Studio 2013 Update 3 with Azure SDK 2.4 added Visual Studio Tooling support to publish WebJobs to Azure Websites. For more information, see How to Deploy Azure WebJobs to Azure Websites

Known Issues when migrating from 0.6.0-beta to 1.0.0-rc1

FileAccess parameter is required when binding blobs to Stream

Starting with this release the SDK requires that FileAccess parameter needs to be set when using Blob attribute and binding to Stream.

public static void BindingToBlob(
[BlobTrigger("container/input")]  Stream input,
[Blob("container/output", FileAccess.Write)]  Stream output
)
{

}

Binding to Azure storage SDK such as CloudBlockBlob or CloudPageBlob or ICloudBlob

Default FileAccess is ReadWrite instead of Read and that is the only available option.

Give feedback and get help

The WebJobs feature of Microsoft Azure Web Sites and the Microsoft Azure WebJobs SDK are in preview. Any feedback to improve this experience is always welcome.

If you have questions that are not directly related to the tutorial, you can post them to the Azure forum, the ASP.NET forum, or StackOverflow.com. Use #AzureWebJobs SDK for Twitter and the tag Azure-WebJobsSDK for StackOverflow.