On June 27, 2018 we announced the preview of Azure Data Lake Storage Gen2 the only data lake designed specifically for enterprises to run large scale analytics workloads in the cloud. Azure Data Lake Storage Gen2 takes core capabilities from Azure Data Lake Storage Gen1 such as a Hadoop compatible file system, Azure Active Directory and POSIX based ACLs and integrates them into Azure Blob Storage. This combination enables best in class analytics performance along with Blob Storage’s tiering and data lifecycle management capabilities and the fundamental availability, security and durability capabilities of Azure Storage.

In this blog post, we are going to drill into why Azure Data Lake Storage Gen2 is unique. Taking a closer look at the innovative Hadoop file system implementation, Azure Blob Storage integration and a quick review of why Azure Data Lake Storage Gen2 enables the lowest total cost of ownership in the cloud.

High-Fidelity server side Hadoop file system with hierarchical namespace

The simplest way to enable Hadoop applications to work with cloud storage is to build a Hadoop File System driver which runs client-side within the Hadoop applications as illustrated in diagram 1. This driver emulates a file system, converting Hadoop file system operations into operations on the backend of the respective platforms. This is often inefficient, and correctness is hard to implement/achieve. For instance, when implemented on an object store a request by a Hadoop client to rename a folder (a common operation in Hadoop jobs) can result in many REST requests. This is because object stores use a flat namespace and don’t have the notion of a folder. For example, renaming a folder with 5,000 items in it can result in 10,000 REST calls from the client . This means 5,000 to copy the child objects to a new destination and 5,000 to delete the original files. This approach performs poorly, often affecting the overall time a job takes to complete. It is also error prone because the operation is not atomic and a failure at any point will result in the job failing with data in an inconsistent state.

In diagram 2 we illustrate how Azure Data Lake Storage Gen2 moves this file-system logic server side along-side our Blob APIs, enabling the same data to be accessed via our BLOB REST APIs or the new Azure Data Lake Storage Gen2 file system APIs. This enables file system operation like a rename to be performed in a single operation. Server-side we are still mapping these requests to our underlying Blob Storage and its flat namespace making this approach more optimal than the first model and offering higher fidelity with Hadoop for some operations, however it doesn’t enable atomicity of these operations.

In addition to moving the file system support server side, we have also designed a cloud scale hierarchical namespace that integrates directly in the Azure Blob Storage. Diagram 3 shows how once enabled the hierarchical namespace provides first class support for files and folders including support for atomic operations such as copy and delete on files and folders. Namespace functionality is available to both Azure Data Lake Storage Gen2 and Blob APIs allowing for consistent usage across both set of APIs. By general availability the same data will be accessible using both BLOB and Azure Data Lake Storage Gen2 APIs with full coherence.

Azure Data Lake Storage Gen2 is building on Blob Storage’s Azure Active Directory integration (in preview) and RBAC based access controls. We are extending these capabilities with the aid of the hierarchical namespace to enable fine-grained POSIX-based ACL support on files and folders. Azure Active Directory integration and POSIX-based ACLs will be delivered during the preview.

Azure Blob Storage integration

Because Azure Data Lake Storage Gen2 is integrated into the Azure Storage platform, applications can use either the BLOB APIs or Azure Data Lake Storage Gen2 file system APIs for accessing data. BLOB APIs allow you to leverage your existing investments in BLOB Storage and continue to take advantage of the large ecosystem of first and third party applications already available while the Azure Data Lake Storage Gen2 file system APIs are optimized for analytics engines like Hadoop and Spark.

Additional benefits from integration with Azure Storage include:

• Unlimited scale and performance due to significant advances made in storage account architecture.
• Performance improvements when reading and writing individual objects resulting in significantly higher throughput and concurrency.
• Removes the need for customers to have to decide a priority whether they want to run analytics or not at the data ingestion time. In Azure Storage we believe all data can and will be used for analytics.
• Data protection capabilities including all data being encrypted at rest using either Microsoft or customer manager keys.
• Integrated network Firewall capabilities that allow you to define rules restricting access to only requests originating from specified networks.
• Durability options such as Zone Redundant Storage and Geo-Redundant Storage to enable your applications to be designed for high-availability and disaster recovery.
• Linux integration - BlobFUSE allows customers to mount Blob Storage from their Linux VMs and interact with Azure Data Lake Storage Gen2 using standard Linux shell commands.

Of course, Azure Storage is built on a platform grounded in strong consistency guaranteeing that writes are made durable before acknowledging success to the client. This is critically important for big data workloads where the output from one task is often the input to the next job. This greatly simplifies development of big data applications since they do not have to work around issues that surface with weaker consistency models such as eventual consistency.

Minimize total cost of ownership

Azure Data Lake Storage Gen2 delivers cloud scale HDFS compatible support optimized for big data workloads such as Hadoop and Spark. With an integrated hierarchical namespace, you now have all the advantages of a file system with the scale and cost typically only associated with object stores. In addition to Blob storage capacity pricing, Azure Data Lake Storage Gen2 also allows you to reduce the cost of your analytics jobs by enabling analytics engines to run more efficiently using a combination of improved storage performance and the ability to perform complex file system operations in a single operation. Azure Data Lake Storage Gen2 also helps you to minimize the total cost of ownership of data over the lifetime of that data using a combination of Blob storage capacity prices, integrated tiering, and lifecycle management policies that enable your data to be tiered from hot to cool to the lowest priced archival solution in the cloud.

Getting started evaluating Azure Data Lake Storage Gen2

To find out more you can:

• Begin your evaluation of Azure Data Lake Storage and sign up for the preview.
• Watch this video to learn how you can get started with Azure Data Lake Storage Gen2.
• Learn more about key capabilities and features of Azure Data Lake Storage Gen2.
• ADLS Gen2 is now available on HDInsight as well. Read more about this integration along with the updates to Hadoop, Spark, and other open source frameworks.

As it’s often the case with new technologies, Augmented Reality (and its more evolved sibling - Mixed Reality)  have attracted a lot of excitement in the tech world and sparked the imaginations of many entrepreneurs. This excitement is easy to understand. This is, after all, a technology that lets us blend the digital with the physical and enhance our senses with super-human capabilities.

However, the value of AR/MR was not immediately obvious to everyone in Manufacturing (we are a skeptical bunch). Early examples of AR/MR applications did not help its case: they showed a group of engineers gathered around a 3D model, waving their hands in the air to virtually pull parts apart and mark things up with “please review this part” virtual notes. The problem with this was that these examples were essentially the same scenarios already beaten to death by VR, with the only apparent difference that engineers used see-through glasses instead of opaque head-mounted devices. So, naturally, people wondered what the big whoop around AR/MR was.

But soon enough folks in the industry realized that the real value of AR/MR was in augmenting the real world with digital information, and not just projecting 3D models in physical space. In other words, providing additional context around the thing you were looking at. That’s when AR/MR “clicked” and more innovative and useful applications started to emerge, now attracting the attention of even the most skeptical and crusty manufacturers.

These AR/MR applications are changing the face of manufacturing: in the plant floor, the field, and in design departments. It’s rapidly becoming the new user interface for factory personnel, service technicians, and engineers.

We can group these emerging applications into two broad categories:

1. Applications that overlay digital information on top of physical things. Information such as:

• Operation, maintenance, and quality control instructions: to guide machine operators, service technicians, and quality assurance staff through the steps needed to perform a task (assemble a component, service a machine, inspect a part). One popular application of this - in an industry hungry for solutions to address an acute skills gap - is training.

• Real-time IoT data: which can help plant workers to quickly identify issues with a machine or process in the plant floor, or a service technician to better service a piece of equipment. For example, thyssenkrupp’s service technicians use Microsoft HoloLens to visualize real-time data about the health of their elevators, which allows them to make better informed decisions on how to best service the units.

• 3D models: of assets and equipment, to preview how they will fit in the building (without the heavy lifting). Microsoft recently released an MR application called Microsoft Layout to do just that. ZF Group, a German car-parts manufacturer, is using Microsoft Layout to help them layout equipment on its factory floor.

2. Applications that enable remote collaboration. For example:

• Remote expert assistance: this is in my opinion one of the killer applications of AR/MR. Here, a technician can collaborate remotely with an expert with heads-up, hands-free video calling, image sharing, and mixed reality annotations. Microsoft also recently released an application to enable this scenario, called Microsoft Remote Assist. ZF Group, mentioned earlier, is also using Remote Assist to connect with remote experts to help them troubleshoot and fix problems with their production machines in hours instead of days that would be required if they had to fly someone in.

• Design reviews/immersive design: AR/MR is a better option than VR for design in those cases where it is important to design the product in context of its physical environment. For example, thyssenkrupp designs custom-built stair lift home solutions. With Microsoft HoloLens and Azure, sales technicians share measurements with remote design teams, and show customers how the stair lift will look, all in one visit.

Getting started with Microsoft AR/MR

You can get started developing exciting AR/MR applications with Microsoft right now! I recommend the following steps:

1. Read the general overview of Mixed Reality. 
2. The Mixed Reality Development Launchpad is your starting point for development. Start with the “Get started” section. 
3. Complete a few tutorials from the Mixed Reality Academy. 
4. If you are interested in Microsoft’s Mixed Reality apps, check out Microsoft Layout and Microsoft Remote Assist. 
5. Learn more about how thyssenkrupp is using HoloLens and Azure IoT Suite.  

Today, we are happy to announce public preview support for autocomplete in Azure Search, one of our most requested features on UserVoice. Autocomplete, also called “type-ahead search”, can enhance the search experience by finding potential terms in the index that match the partial term being written by the user. An example of similar functionality is shown below from bing.com:

Autocomplete modes

For autocomplete to work, the fields in the search index should be listed in a suggester. For example, if you define a suggester on a city field, typing "sea" will return terms like “Seattle” or “Seaside” if those terms exist in the city field.

The table below describes the 3 modes supported by Autocomplete and the expected results in each scenario.

Autocomplete API also supports fuzzy matching in each of these modes. When fuzzy is enabled, it compensates for typos and misspellings, and proposes relevant results based on user intent. For more details about these options, please refer the API documentation.

Suggest vs. Autocomplete

For those of you already familiar with Azure Search, you might be wondering how Autocomplete compares to the existing Suggestions feature. Given a partial query term, Suggestions API suggests documents and returns document Ids which contain the query term, while the Autocomplete API returns potential terms from the index which match the partial term in the query. The following examples illustrate the same:

Suggested results

Autocompleted results

Next steps

• Try out these new enhancements using the tutorial.
• Learn more by visiting documentation for SDK and API.

Please note that the API is available in the 2017-11-11-Preview version. If you have already defined a suggester on some fields in your index, you can start experimenting with this API right away. For everybody else, you'll need to add fields and bind them to a new suggester, or create a new index with fields included in a suggester. You can't add a suggester that includes fields without recreating your index.

If you are already using Suggest from Azure Search, please give this a try. We’d love to hear your feedback!

Insurers need to become the home page for their customers. A home page is a source of meaningful, targeted and useful content that the customer seeks out. Technology can now deliver innovative solutions directly to insurance customers. Artificial Intelligence (AI) and Machine Learning (ML) are delivering targeted content continually from multiple sources, including data generated from IoT devices owned by customers, in real time. Focusing on the customer will make the relationship more valuable to the consumer, and will increase the number of interactions with the insurance company. Technology is making that happen from the point of first interaction. For example, when a potential customer interacts with a company’s website, or with an app powered by bots. Then the company uses AI to make sure the products are correctly matched and priced for the customer. As the needs of the customer changes, the enterprise delivers continuing, direct support.

A survey by InsuranceNexus in June 2018, titled Insurance Customer Engagement Europe, 400 senior insurance executives were surveyed, 91 percent agreed that the insurance industry must become more customer-centric. With that, Bain reports that insurance companies can boost revenues, improve margins and most importantly, sustain the loyalty of their very demanding customers. As shown in this graphic, “offering services beyond insurance leads to higher customer loyalty”.

Top InsurTech influencers like Matteo Carbone, Dr. Robin Kiera, and Miguel Edwards (CIO at Kemper), are promoting these ideas at InsurTech events. At InsurTech Rising in Boston, Miguel Edwards used a real example of an insurance company selling pet insurance with a range of other services. Figo offers “your pet’s life in the palm of your hand.” It combines record management of medical, adoption/ownership, billing, and licenses/registration documents with tools that manage pet photos and social contacts, It also sells pet health insurance. The added value becomes the selling point to the consumer and a data collection point for risk assessment. The company offers free pet tags to identify lost pets. It also offers a GPS-enabled collar for real-time tracking (with text alerts) if a pet goes outside a preset area. Other services include health reminders, pet activities, and events, as well as pet social media connections. These additional services increase value to the customer and build brand loyalty.

For the insurance industry, the key is to interact with customers, add value and build trust. The traditional behavior is that interaction occurs only when a customer buys a policy, and again when a claim occurs. Now an insurance company is a real-time risk evaluator and can prevent or mitigate claim events, and be a valuable data source that is important to the customer. Technology changes the process. Chat bots, driven by AI, direct customers to the most appropriate products and price the products based on the most current risk assessment. The customer interaction continues afterwards, IoT and telemetric data are used to adjust risk then alert customers of claim-generating events.

Insurance industry leader, John Hancock, uses wearable devices connected to an application called Vitality. The company can offer discounts on life insurance policies based on the customer's activity as tracked by the Fitbit or Apple Watch. Vitality is also part of a health and wellness program with information on nutrition and active lifestyle choices. It is also used to promote the benefits of increased physical activities to its employees with a net increase in productivity, employee satisfaction, and reduction sick leave usage. The use of products like Vitality is becoming the norm for major employers to offer incentives to employees based on wearable device activity, from cash rewards to special recognition.

Thought leaders in InsurTech are pushing the insurance industry to be more customer-focused. With the increased use of IoT data to track changes in risk, it gives insurers the opportunity to be more valuable to their customers. The use of AI at all levels of the insurance value chain will ensure customers are correctly matched with the right insurance products, and made aware of any gaps in coverage in real-time.

Learn how to create personalized experiences for your customers by reading this overview on the Azure Bot Service and then create your first bot using the service. 

For those of you who might not have tried it yet, Video Indexer is a cloud application and platform built upon media AI technologies to make it easier to extract insights from video and audio files. As a starting point for extracting the textual part of the insights, the solution creates a transcript based on the speech appearing in the file; this process is referred to as Speech-to-text. Today, Video Indexer’s Speech-to-text supports ten different languages. Supported languages include English, Spanish, French, German, Italian, Chinese (Simplified), Portuguese (Brazilian), Japanese, Arabic, and Russian.

However, if the content you need is not in one of the above languages, fear not! Video Indexer partners with other transcription service providers to extend its speech-to-text capabilities to many more languages. One of those partnerships is with Zoom Media, which extended the Speech-to-text to Dutch, Danish, Norwegian and Swedish.

A great example for using Video Indexer and Zoom Media is the Dutch public broadcaster AVROTROS; who uses Video Indexer to analyze videos and allow editors to search through them. Finus Tromp, Head of Interactive Media in AVROTROS shared, “We use Microsoft Video Indexer on a daily basis to supply our videos with relevant metadata. The gathered metadata gives us the opportunity to build new applications and enhance the user experience for our products.”

Below is an example for how a Dutch video transcript looks like in the Video Indexer Service:

Speech-to-text extensions in Video Indexer

To index a file in one of the extended language you will need to first use the provider’s solution to generate a VTT file (aka the result transcription file) in the required language and then send the VTT file along with the original video or audio file to Video Indexer to complete the indexing flow.

Once the file is indexed, the full set of insights, including the full transcript will be available for consumption both via the Video Indexer API and via the Video Indexer service. Video Indexer also include the ability to translate the that transcript to dozens of other languages.

We are excited to introduce an open-sourced release of an integration between Video Indexer and Zoom Media now available on GitHub.

(Special thanks to Victor Pikula, Cloud Solution Architect in Microsoft, for designing and building this integration)

The solution is built using Azure Blob storage, Azure Logic Apps and the new v2 Video Indexer REST API. Let’s take a deeper look into how the integration is built.

Below is a high-level diagram of the flow:

 
Given a video or audio file, the file is first dropped into a Blob Storage. The Logic App “watches” any additions to the blob and as a result sends the file to both Video Indexer and Zoom Media. Zoom Media then generates a VTT file based on the required language and passes it back to Video Indexer to complete the indexing flow. There is more information about the integration, how to get Video Indexer key and the Zoom Media key, and how to control the VTT language used available.

Having the integration built using Logic Apps makes it easy to customize and maintain it as it allows you to debug and configure it quickly and without any need of coding skills.

 
So, as you can see, with Video Indexer’s powerful media AI technologies, coupled with the ability integrate speech-to-text capabilities in any language and the ease of use of Azure services such as Logic Apps, everyone can get more out of their videos around the world!

Have questions or feedback? We would love to hear from you!

Use our UserVoice to help us prioritize features, or email VISupport@Microsoft.com with any questions.

This post was authored by Jason Haley, Microsoft Azure MVP.

Recently, I was at Boston Code Camp catching up with some old friends and looking to learn about containers or anything that could help me in my current project of migrating a microservices application to run in containers. I was speaking with one friend who had just presented a session on Polly, and he made a comment that got my attention. He said that one of the attendees at his session was under the impression that using the cloud would make his application inherently resilient and he would not need any of the features that Polly provides.

In case you are not familiar with Polly, you can use this library to easily add common patterns like Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback to your code to make your system more resilient. Scott Hanselman recently wrote a blog post: Adding Resilience and Transient Fault handling to your .NET Core HttpClient with Polly, discussing how he was using Polly and HttpClient with ASP.NET Core 2.1.

What that attendee may have been referring to is that most Azure services and client SDKs have features to perform retries for you (which can play a large part in making your application resilient), but in some cases you need to specifically set up the retry logic. Also keep in mind that your third-party client SDKs may need retry logic turned on in diverse ways. Your application will not automatically become resilient just by putting it in the cloud.

What is resiliency?

Resiliency is the capability to handle partial failures while continuing to execute and not crash. In modern application architectures — whether it be microservices running in containers on-premises or applications running in the cloud — failures are going to occur. For example, applications that communicate over networks (like services talking to a database or an API) are subject to transient failures. These temporary faults cause lesser amounts of downtime due to timeouts, overloaded resources, networking hiccups, and other problems that come and go and are hard to reproduce. These failures are usually self-correcting.

You can’t avoid failures, but you can respond in ways that will keep your system up or at least minimize downtime. For example, when one microservice fails, its effects can cause the system to fail.

With modern application design moving away from the monolithic application toward microservices, resiliency becomes even more important due to the increased number of components that need to communicate with each other.

How can you make your system more resilient?

Lately I’ve been reading another great resource, the .NET Microservices: Architecture for Containerized .NET Applications e-book, which also has a reference microservice application on GitHub called eShopOnContainers. Written in .NET Core 2.1, eShopOnContainers is a microservices architecture that uses Docker containers. I’ll be referring to the reference application for sample code. Keep in mind, there are some new technologies that promise to help make service-to-service communication more resilient without adding code, like a service mesh, which I won’t be discussing here. I want to look at what type patterns we can use in code. Let’s look at examples of a couple of resiliency patterns: Retry and Circuit Breaker. The Azure Architecture Center covers several resiliency patterns that you can use in your application.

Retry

Retries can be an effective way to handle transient failures that occur with cross-component communication in a system. As I mentioned, most Azure services and client SDKs have features for performing retries. One example when working with a database is to use Entity Framework Core and EnableRetryOnFailure to configure a retry strategy. In the eShopOnContainers code, you can see an example of this by looking at the Startup.cs file in the Catalog.API project. In the AddCustomDbContext method of the CustomExtensionsMethods utility class towards the bottom where it configures CatalogContext, you’ll see the usage of EnableRetryOnFailure.

The code above shows Entity Framework Core is to retry database calls up to 10 times before failing and to add some time delay between retries — but not delay more than 30 seconds. This is using the default execution strategy (there are others). If you want to know more about configuring Entity Framework Core to automatically retry failed database calls, you can find the details at Connection Resiliency.

Circuit Breaker

Developers often use the Circuit Breaker and Retry patterns together to give retrying a break. Retry tries an operation again, but when it doesn’t succeed, you don’t always want to just try it one more time or you may risk prolonging the problem (especially if the failure is due to a service being under a heavy load). The Circuit Breaker pattern effectively shuts down all retries on an operation after a set number of retries have failed. This allows the system to recover from failed retries after hitting a known limit and gives it a chance to react in another way, like falling back to a cached value or returning a message to the user to try again later.

Earlier this year when I read the .NET Microservices: Architecture for Containerized .NET Applications e-book, I found the especially useful HttpClient wrapping class named ResilientHttpClient in the chapter on resiliency. This utility class was the reason that I cloned the GitHub repo and started learning the eShopContainers code. Following the codebase update to use .NET Core 2.1, refactoring removed that utility class in favor of using new features that do the same thing. But if you are curious or are not using .NET Core 2.1 yet, you can find the code in ResilientHttpClient.cs on GitHub. The HttpInvoker method is the heart of this utility.

This method uses Polly to make a call using an HttpClient with an exponential back-off Retry policy and a Circuit Breaker policy that will cause retries to stop for a minute after hitting a specified number of failed retries. The last line in the method is the one that makes the call by executing the passing in action. This may not make a lot of sense just from the small code snippet, but Section 8 of the e-book, Implementing Resilient Applications, goes into detail on how the ResilientHttpClient utility class works. The polices are in the CreateResilientHttpClient method of the ResilientHttpClientFactory class:

You should be able to understand what the policies are, but again I refer you to the e-book for a detailed explanation. Also, if you are using .NET Core 2.1, the e-book’s update has useful information on how to configure and use the new HTTPClientFactory to do the same thing.

Summary

Running your application in containers or in the cloud does not automatically make your application resilient. It’s up to you to configure the features that will enable the retry logic you provide. When you need retry logic added to your system, you should use a library such as Polly to speed up your implementation. Or, if you are exploring how to add resiliency without code, you should investigate service mesh products like Istio and Linkerd.

If you use HttpClient in your applications to call APIs, you should download the .NET Microservices: Architecture for Containerized .NET Applications e-book and clone the GitHub repo. The e-book discusses the reference architecture in depth to help you understand microservices architecture.

There’s also this recording of an Ignite 2017 breakout session about the e-book and eShopOnContainers project: Implement microservices patterns with .NET Core and Docker containers.

The digital experience (DX) era is here, and AI is one of the primary technologies to fuel productivity and innovation in the retail and consumer goods industry. Brands that take a wait and see approach may find themselves quickly outpaced by their competitors. And by competitors, I mean not just born-in-the-cloud e-commerce players, but forward-thinking omnichannel retailers who focus on winning customers and evolving retail at scale.

Within the spectrum of digital transformation, AI is not a new technology. It is moving from its research roots to entering the mass market. This is made possible by the growth of cloud computing, availability of big data, and years of improved algorithms developed by researchers. At its core, AI gives computers decision-making capabilities to solve problems in a more natural and responsive way, as compared to the practice of pre-programmed computer routines of today. AI will be an imperative for optimization, automation, scale and most importantly, (gulp) survival.

Cloud computing and big data are accelerating AI technology

According to Accenture, unlimited access to computing power and the growth in big data are creating the right environment for AI. Analyzing data requires massive compute and storage; the cloud provides an efficient way to run AI systems. A general consensus is that public cloud spending is increasing at a growing rate every year. For example, Forbes states that the “total public cloud market will be $178 billion in 2018, up from $146 billion in 2017, growing at a 22 percent compound annual growth rate.” Further, the digitization of everyday life is enabled by the proliferation of devices and the cost effectiveness of sensors. That in turn has created massive amounts of data available for computers to learn from. Deep learning and computer vision will thrive on the accessible data. By 2025, big data is estimated to grow to 180 zettabytes, up from less than 10 zettabytes of data in 2015.

The industry is placing bets on AI

A key factor in this discussion is the value creation attributed to AI technology. According to a McKinsey study, it is estimated that AI could potentially generate $4 billion to $8 billion in annual value for the global retail economy, and $2 billion to $5 billion in consumer goods. According to a study on Forbes, 81 percent of IT leaders are currently testing, or planning to invest, in AI.

According to angel.co, since July 2017, there are 4,701 AI startups, up from 2,200 AI startups. Machine Learning, Natural Language Processing, and Computer Vision startups make up the top three categories within AI technologies. This signals the anticipation of demand, and the opportunity to transform.

Customer-centric technology that responds using natural language and behavior

The technologies that define AI fall into four categories, sense, learn, reason and act. According to the Microsoft e-book, The Future Computed, these capabilities will enable computers to more naturally interact in the world around them through technologies that enable vision, speech, language and knowledge.

The power of leveraging these capabilities in combination can make an experience with a customer more natural and seamless. On a macro level, AI technology lends itself to functions within an organization with massive amounts of structured and unstructured marketing, sales, logistics, operations, finance, and call center data. If you go a few clicks deeper, use case domain examples include: data-driven accounting, data-driven hiring, lead generation, fraud and debt analysis, inventory optimization, pricing and promotion, allocation, demand forecasting, personalized recommendations, search, and more. These examples illustrate opportunities to use AI to improve existing analytics use cases. Per McKinsey research, potential for incremental value from AI over other analytics techniques is 87 percent and 55 percent for retail and consumer goods, respectively.

I’d be overzealous if I didn’t acknowledge that there are pre-requisites to leveraging AI in today’s organizations. Obstacles include data silos and legacy technology. To produce value, and leverage AI’s full potential, you must have data to process. The state is not as important as long as it can be processed and remain within regulatory compliance in the cloud. In other cases, data may not be captured even though it is generated, so again, the pre-requisite is having available data to process. Now is the time to get an organization’s data estate in order. We’ve seen this taking shape with efforts such as Customer 360, Unified Commerce, Order Management, etc. where data is the backbone of the effort.

The democratization of AI is underway

Microsoft’s AI platform provides core capabilities through a common set of APIs, and tools for practitioners who want to be hands-on in the creation of their own custom AI models. They can do so leveraging an AI foundation built on ethical principles to ensure AI helps all people.

For organizations looking to get started, Microsoft offers pre-trained AI services with pre-built AI models. These include Azure-powered Cognitive Services to infuse apps with intelligent algorithms to see, hear, speak, understand and interpret your user needs through natural methods of communication. Conversation AI integrates naturally fluid conversation into the consumer experience. Azure Bot Service and LUIS can aid in creation, deployment, and management of a digital assistant that allows consumers access to your brand 24/7. 

AI services and tools are also integrated into the Azure Portal and Visual Studio. There are tools for developers to train and deploy their own models such as TensorFlow, and Cognitive Toolkit to name a few. Lastly, Microsoft is invested in solutions where AI can address industry-specific use cases. There are options from build to buy, first-party or partner-led to engage with Microsoft’s AI technology.

Where to start

Learn best practices for managing retail data by reading this overview on Data Management. Visit Azure AI to learn more about Microsoft’s AI tools and services, including the opportunity to try for free Cognitive Services that infuse your customer touch points with natural methods of communication.

What other opportunities and challenges do you see with AI and cloud computing? This is just the tip of the iceberg for retail and consumer goods. I welcome your feedback and questions.

Today we are excited to announce the public preview of static website hosting for Azure Storage! The feature set is available in all public cloud regions with support in government and sovereign clouds coming soon.

This set of new storage capabilities enables a cost-effective and scalable solution for hosting integrated modern web applications on Azure. On a static website, webpages contain static content and JavaScript or other client-side code. By contrast, dynamic websites depend on server-side code, and can be hosted using Azure Web Apps. As deployments shift toward elastic, cost-effective models, the ability to deliver web content without the need for server management is critical. The introduction of static website hosting in Azure Storage makes this possible, enabling rich backend capabilities with serverless architectures leveraging Azure Functions and other Paas Services.

How it works

When you enable static websites on your storage account, a new web service endpoint is created of the form

<account-name>.<zone-name>.web.core.windows.net.

The web service endpoint always allows anonymous read access, returns formatted HTML pages in response to service errors, and allows only object read operations. The web service endpoint returns the index document in the requested directory for both the root and all subdirectories. When the storage service returns a 404 error, the web endpoint returns a custom error document if you configured it.

Getting started

This feature set is supported by the most recent releases of the Azure Portal, .Net Client Library (version 9.3.0), Java Client Library (version 8.0.0), Python Client Library (version 1.3.0), Node.js Client Library (version 2.10.0), Visual Studio Code Extension (version 0.4.0), and CLI 2.0 (extension version 0.1.3). Support for Storage Explorer and Powershell are coming soon. You can also directly use the Storage Services REST API as always. Static websites is supported by REST API version 2018-03-28 and greater. In general, we always recommend using the latest version regardless of whether or not you are using this feature.

To start hosting your web application on Azure Storage, you can configure the feature using the Azure Portal as shown in the screenshot below. Create a GPv2 storage account in the Azure Portal or use an existing GPv2 account and click on static website (preview) under settings in the left navigation bar. Click Enabled and enter the name of the index document and the optional custom error document path.

Upload your web assets to the $web container that was created as a part of static website enablement. You can do this directly in Azure Portal, or you can take advantage of the Visual Studio Code Extension to upload entire directory structures. Right click on the $web container, select Deploy to static website and find the directory you want to upload. Make sure to include an index document with the name you configured. In this example, the document's name is index.html.

Finally, navigate to your web endpoint to test your website. You can also check out this step-by-step tutorial on building your own serverless web app for guidance on adding richer capabilities to your website.

Get it, use it, and tell us about it

Static website hosting support in Azure Storage is provided at no additional cost. For more details on the feature set, check out the static websites documentation.

We look forward to hearing your feedback on this feature through email at AzureStorageFeedback@microsoft.com. As a reminder, we love hearing all of your ideas and suggestions about Azure Storage, which you can post at Azure Storage feedback forum.

Why lambda?

Lambda architecture is a popular pattern in building Big Data pipelines. It is designed to handle massive quantities of data by taking advantage of both a batch layer (also called cold layer) and a stream-processing layer (also called hot or speed layer).

The following are some of the reasons that have led to the popularity and success of the lambda architecture, particularly in big data processing pipelines.

Speed and business challenges

The ability to process data at high speed in a streaming context is necessary for operational needs, such as transaction processing and real-time reporting. Some examples are fault/fraud detection, connected/smart cars/factory/hospitals/city, sentiment analysis, inventory control, network/security monitoring, and many more.

Typically, batch processing, involving massive amounts of data, and related correlation and aggregation is important for business reporting. This is to understand how the business is performing, what the trends are, and what corrective or additive measure can be executed to improve business or customer experience.

Product challenges

One of the triggers that lead to the very existence of lambda architecture was to make the most of the technology and tool set available. Existing batch processing systems, such as data warehouse, data lake, Spark/Hadoop, and more, could deal with petabyte scale data operations easily but couldn’t do it fast enough that was warranted by the operational needs.

Similarly, very fast layers such as cache databases, NoSQL, streaming technology allows fast operational analytics on smaller data sets but cannot do massive scale correlation and aggregation and other analytics operations (such as Online Analytical Processing) like a batch system can.

The skills challenge

Additionally, in the market you will find people who are highly skilled in batch systems, and often they do not have the same depth of skills in stream processing, and vice versa.

Lambda on Azure

The following is one of the many representative Lambda architecture on Azure for building Big Data pipelines.

 
Figure 1: Lambda architecture for big data processing represented by Azure products and services. Note, other Azure and (or) ISV solutions can be placed in the mix if needed based on specific requirements.

What problems do lambda solve vs. what problems does it introduce?

As stated in the previous section, lambda architecture resolves some business challenges. Various parts of the business have different needs in terms of speed, level of granularity and mechanism to consume data. It also resolves the challenge of the choice of technology, by using the best of the speed layer and batch layer together, and not stretching one product to do both which it isn’t comfortable in doing. Finally, it ensures people with skills dealing with transaction and speed layer can work in parallel and together with people with skills in batch processing.

Although immensely successful and widely adopted across many industries and a defacto architectural pattern for big data pipelines, it comes with its own challenges. Here are a few:

• Transient data silos: Lambda pipelines often creates silos that could may cause some challenges in the business. The reporting at the speed layer that the operations team is dealing with, may be different for the aggregate batch layer that the management teams are working with. Such creases may eventually iron out, but it has the potential of causing some inconsistencies.

• More management overhead: It also increases the number of subsystems, as a result during maintenance time, many needed to be managed and maintained. This could potentially mean one may need bigger teams with deep and wide skill sets.

The emerging big data design pattern

If there was a way that utilized the right mix of technologies that didn’t need a separate speed or batch layer, we could build a system that has only a single layer and allows attributes of both the speed layer and batch layer. With the technological breakthrough at Microsoft, particularly in Azure Cosmos DB, this is now possible.

Azure Cosmos DB is a globally distributed, multi-model database. With Cosmos DB you can independently scale throughput and storage across any number of Azure's geographic regions. It offers throughput, latency, availability, and consistency guarantees with comprehensive service level agreements (SLAs).

Here are some of the key features that renders Cosmos DB as a suitable candidate for implementing the proposed reference architecture where the speed later and the batch layer merges into a single layer.

Cosmos DB change feed

• Most importantly, the key feature that is pivotal in building this emerging big data architectural pattern is the Cosmos DB change feed. Change feed support works by listening to an Azure Cosmos DB collection for any changes. Then, it outputs the sorted list of documents that were changed in the order in which they were modified. The changes are persisted, can be processed asynchronously and incrementally, and the output can be distributed across one or more consumers for parallel processing.

Features for speed/hot layer

• Azure Cosmos DB’s database engine is fully schema-agnostic – it automatically indexes all the data it ingests without requiring any schema or indexes and serves blazing fast queries.
• Cosmos DB allows you to easily scale database throughput at a per-second granularity, and change it anytime you want.
• You can distribute your data to any number of Azure regions, with the click of a button. This enables you to put your data where your users are, ensuring the lowest possible latency to your customers.
• Azure Cosmos DB guarantees end-to-end low latency at the 99th percentile to its customers. For a typical 1KB item, Cosmos DB guarantees end-to-end latency of reads under 10 ms and indexed writes under 15 ms at the 99th percentile, within the same Azure region. The median latencies are significantly lower (under 5 ms).

Features for batch/cold layer

• You can access your data by using APIs of your choice, like the SQL, MongoDB, Cassandra API, and Table APIs, and graph via the Gremlin API. All APIs are all natively supported.
• You can also scale storage size transparently and automatically to handle your size requirements now and forever.
• Five well-defined, practical, and intuitive consistency models provide a spectrum of strong SQL-like consistency all the way to the relaxed NoSQL-like eventual consistency, and everything in-between.
• Rapidly iterate the schema of your application without worrying about database schema and/or index management.
• Using the features described above, the following will be an implementation of the emerging architectural pattern.

The following is a diagrammatic representation of the emerging big data pipeline that we have been discussing in this blog:

Figure 2: Emerging architectural pattern implemented using Cosmos DB for Big Data pipelines as an evolution of the traditional lambda architecture.

Hence, by leveraging Cosmos DB features, particularly the change feed architecture, this emerging pattern can resolve many of the common use-cases. This in turn, gives all the benefits of the lambda architecture, and resolves some of complexities that lambda introduces. More and more customers adopting this and resulting in a successful community, and success of this new pattern and increased adoption of Azure Cosmos DB.

What’s next?

The following is a list of resources that may help you get started quickly:

• If you haven't already, download the Spark to Azure Cosmos DB connector from the azure-cosmosdb-spark GitHub repository.
• The stream feed from Twitter to CosmosDB, which is the mechanism to push new data into Azure Cosmos DB.
• As well with the Cosmos DB Time-to-Live (TTL) feature, you can configure your documents to be automatically deleted after a set duration. For more information on the Azure Cosmos DB TTL feature, see Expire data in Azure Cosmos DB collections automatically with time to live.
• Azure Cosmos DB BulkExecutor library overview.
• Hands on labs using Azure Services to build Azure IoT End to End Solutions connecting real and simulated devices to Azure IoT Hub.
• Perform graph analytics by using Spark and Apache TinkerPop Gremlin.
• Connect to Azure Cosmos DB using BI analytics tools with the ODBC driver.

Bing has made very significant investments in the area of deep learning, computer vision and artificial intelligence to support a number of different search scenarios. The Bing Maps team has been applying these techniques as well with the goal to increase the coverage of building footprints available for OpenStreetMap. As a result, today we are announcing that we are releasing 124 Million building footprints in the United States to the OpenStreetMap community.

The Maps team has been relying on the Open Source CNTK Unified Toolkit which was developed by Microsoft. Using CNTK we apply our Deep Neural Networks and the ResNet34 with RefineNet up-sampling layers to detect building footprints from the Bing imagery.

We remove noise and suspicious data (false positives) from the predictions and then apply a polygonization algorithm to detect building edges and angles to create a proper building footprint.

Using this approach we extracted 124,885,597 footprints in the United States. In OpenStreetMap there are currently 30,567,953 building footprints in the US (the last time our team counted) both from editor contributions and various city or county wide imports. Bing is making this data available for download free of charge.

You can read more about this work and directly download the data from GitHub. The CNTK toolkit developed by Microsoft is open source and available on GitHub as well. The ResNet3 model is open source and available on GitHub. Our Bing Maps computer vision team will be presenting this work at the annual International State of the Map conference in Milan, Italy.

- Bing Maps Team

One of the most common questions I get asked is, "Should I learn R or Python?". My general response is: it's up to you! Both are popular open source data platforms with active, growing communities; both are are highly sought after by employers, and both have a rich set of capabilities for working with data. It really depends most on your interests and the kind of employer you want to work for.

If your interests lean more towards traditional statistical analysis and inference as used within industries like manufacturing, finance, and the life sciences, I'd lean towards R. If you're more interested in machine learning and artificial intelligence applications, I'd lean towards Python. But even that's not a hard-and-fast rule: R has excellent support for machine learning and deep learning frameworks, and Python is often used for traditional data science applications.

One thing I am quite sure of though: neither Python nor R is inherently better than the other, and arguments on that front are ultimately futile. (Trust me, I've been there.) Which is better for any given person depends on a wide variety of factors, and for some, it may even be worthwhile to learn both. Brian Ray recently posted a good overview of the factors that may lead you towards R or Python for data science: their history, the community, performance, third-party support, use cases, and even how to use them together. It's great food for thought if you're trying to decide which community to invest in. The cartoon below is taken from that post, and sums up things quite nicely.

And what about me, you ask? My needs generally fall on the statistics / data science end of the spectrum, and my interest in deep learning has been served well by the keras support from RStudio. I've been using R for a long time, and I just haven't found anything yet that I, personally, has led me down the Python path. But your needs may vary from mine, and the article linked below is a good guide to that decision.

Brian Ray: Python vs (and) R for Data Science

On Wednesday, June 27, 2018, we announced the general availability of Azure IoT Edge. This release adds tons of new features for those already using public preview bits. Customers who have never used Azure IoT Edge can start with the Linux or Windows quickstarts. Those who have started projects on preview bits should upgrade to the latest bits and integrate breaking changes.  Details on both of these processes are below.

Upgrade to the latest bits

Uninstall preview bits

Use iotedgectl to uninstall the preview bits from your Edge device by running the following command. You can skip this step if you are installing GA bits on a device or VM that has never run preview bits.

iotedgectl uninstall

Delete preview runtime container images

Use “docker rmi” to remove the container images for preview versions of Edge Agent and Edge Hub from your Edge device. You can skip this step if you are installing GA bits on a device or VM that has never run preview bits.

Remove references to preview container images in deployments

The IoT Edge Security Daemon includes functionality to allow the user to specify which versions of the Edge Agent and Edge Hub are used by an Edge device. This allows you to control when bits on your device get updated. The version of these runtime components are specified in your deployment. Deployments created during preview include references to preview container images. You must change them to reference GA container images, ensuring that the Security Daemon does not downgrade your machine to preview bits. An example deployment snippet is below.

"systemModules": {
     "edgeAgent": {
         "type": "docker",
         "settings": {
             "image": "microsoft/azureiotedge-agent:1.0-previewmcr.microsoft.com/azureiotedge-agent:1.0",
             "createOptions": "{}"
         }
     },
     "edgeHub": {
         "type": "docker",
         "settings": {
             "image": "microsoft/azureiotedge-hub:1.0-previewmcr.microsoft.com/azureiotedge-hub:1.0",
             "createOptions": "{}"
         },
         "status": "running",
         "restartPolicy": "always"
     }
},

Install GA bits

The quickstarts for Linux and Windows walk you through setting up an IoT Edge device with these new infrastructural components. You will notice that the quickstarts start by installing a component introduced with GA called the IoT Edge Security Daemon. This is part of the Edge Security Manager, a set of components that help us secure the edge.

The IoT Edge Security Daemon starts each time an Edge device boots and initiates the IoT Edge Agent. This means that iotedgectl is no longer needed when working with IoT Edge devices.

Hsmlib is another new component which is part of the IoT Edge Security Manager. Different versions of hsmlib integrate with device specific hardware and allow the IoT Edge Security Daemon to perform cryptographic functions. Advanced types of hardware enable IoT Edge to provide innovative security and functionality. For example, devices with a hardware security module (eg. TPM) can be provisioned via the Device Provisioning Service.

A default version of hsmlib is installed along with the IoT Edge Security Daemon. This version of hsmlib supports our standard security promise which uses the Edge device’s file system for storage of secrets as well as our secure enclave promise using TPM at /tpm0.

Integrate breaking changes

The vast majority of feature additions have been done in a backwards compatible manner making it easy to migrate your projects on to GA bits. We’ve learned an incredible amount from having the product in public preview and are making a few small breaking changes based on user feedback. These changes are as small as possible while still delivering usability improvements to the product.

Client SDK split

Change

The DeviceClient object, used to write IoT Edge modules, will split into two objects: DeviceClient and ModuleClient.

Impact

Small code change. You must change the class name for all DeviceClient objects in your module code to ModuleClient. We've also introduced a factory class for creating ModuleClients. This helper class hides the details of correctly integrating with the IoT Edge Security Daemon.

Before

DeviceClient deviceClient = DeviceClient.CreateFromConnectionString(connStr, transportSettings);

After

ModuleClient moduleClient =
ModuleClient.CreateFromEnvironment(transportSettings);

Additionally, the API surface of the ModuleClient will be reduced to only the functionality that is currently supported for modules. This should not affect any existing module code as those APIs do not work.

Reason for change

The functionality available to devices and modules is slightly different. For example:

• Modules can read/write from inputs/outputs while devices can only write to IoT Hub.
• Devices can receive C2D messages while modules cannot.
• Devices can perform FileUpload while modules cannot.
• Separating the clients cleanly delineates the functionality available to devices and modules.

Direct method invocation

Change

The ServiceClient class will no longer allow module identities to invoke direct methods. This functionality has been moved to the ModuleClient class as part of the Client SDK split detailed above.

Impact

You must use the ModuleClient class to invoke methods for module identities instead of the ServiceClient.

Before

// assumes the use of a module connection
ServiceClient serviceClient = ServiceClient.CreateFromConnectionString(connStr, transportSettings);
string.serviceClient.InvokeDeviceMethodAsync("DeviceId", "ModuleId", new CloudToDeviceMethod("MethodName"));

After

ModuleClient moduleClient = ModuleClient.CreateFromEnvironment(transportSettings);
moduleClient.InvokeMethodAsync("DeviceId", "ModuleId", new DirectMethodRequest("MethodName")) ;

Reason for change

Invoking a method is an action users expect to be able to do with the ModuleClient, instead of instantiating a second object. Adding method invocation to ModuleClient reduces the size of module code as developers no longer need to include the ServiceClient in their code.

Configure port on which Edge Hub listens

Change

Edge Hub used to listen for connections from downstream devices on ports 8883 and 443. Now we require developers to specify on which host ports it uses for these connections.

Impact

You must set port forwarding info on Edge Hub when manually creating a deployment's JSON. The below JSON configures Edge Hub to work as before (listen on ports 8883 and 443).

"{"HostConfig":{"PortBindings":{"8883/tcp":[{"HostPort":"8883"}],"443/tcp":[{"HostPort":"443"}]}}}"

You do not need to make any changes if you're creating a deployment through the Azure portal or with the VS Code extension. These tools will automatically set up port forwarding options to make Edge Hub listen for connections on 8883 and 443.

Reason for change

Some customers may want an Edge device to listen for downsteam device connections on a port other than 443 or 8883 as those ports could be used for a different purpose. For example, customers may want to run a HTTPS server on 443.

This month, we introduced several new capabilities that streamline the management of common tasks and enhance identity-driven security measures to improve and protect the user experience in Microsoft 365.

Streamlining the way you work

Updates to the Office 365 user experience—We announced updates for Word, Excel, PowerPoint, OneNote, and Outlook that are designed to embrace the breadth and depth of Office 365 features, while simplifying the user interface and improving accessibility. These updates include a simplified ribbon to encourage focus and collaboration, modern colors and icons to improve rendering and accessibility, and AI-powered search to quickly surface relevant information. These changes will start to roll out to Microsoft 365 and Office 365 subscribers over the next few months.

Connect Office 365 Groups to SharePoint sites—Office 365 Groups can now connect to existing SharePoint sites, allowing newly created Office 365 groups to integrate with your existing SharePoint infrastructure. Connecting a group to a site provides a single starting point to find content, team news, and communications with modern pages, libraries, and lists—without losing any previous content or permissions.

Reduce distractions with Outlook for Android—We introduced “Do Not Disturb” in Outlook for Android to help you reduce distractions and get more done. Now, subscribers can set timed or scheduled periods when email and calendar notifications will be paused. For those with multiple Outlook accounts, Do Not Disturb settings can be customized for each email address—enabling granular control over how you spend your focus-hours.

Manage progress in Microsoft To-Do—This month, we introduced “Steps” in Microsoft To-Do—a new feature that allows you to break down tasks into smaller, incremental steps—making large projects more manageable. Now, when a you create a To-Do item, you can add a range of detailed steps that are tracked through to completion. We also introduced the ability to share your To-Do lists, enabling you to work together on tasks and complete projects with colleagues and friends.

Dictation in OneNote—Office 365 subscribers with Windows 10 can now take advantage of hands-free dictation using nine languages in OneNote. Dictation provides a simple, yet transformational, way to express ideas and capture notes using only your voice. You can also make edits using your keyboard without having to pause the recording. Simply click or tap the Dictate icon and start speaking.

Adobe PDF integration in Office 365—Last September, we expanded our strategic partnership with Adobe to focus on integrations between Adobe Sign and Office 365 products, like Microsoft Teams, SharePoint, and Outlook. This month, the Adobe Document Cloud team announced new capabilities for OneDrive and SharePoint that provide improved fidelity when working with PDF documents. Once integrated by your administrator, PDF services provide rich previews of PDF documents in OneDrive and your SharePoint sites, and allow you to combine several files into a single PDF in your document library.

Securing the modern workplace

We introduced several new important capabilities that strengthen your organization’s identity-driven security, and ensure important data is kept safe.

Protect identities and accounts with Microsoft Baseline—We introduced a set of policies for privileged accounts called Microsoft Baseline that protect identities and accounts with administrator or other expanded access privileges. Now, customers will be opted in to Baseline protection policies by default with the ability to opt out at any time. These new policies will enforce industry-standard protection measures like Multi-Factor Authentication to secure the most important accounts in your tenant.

Block legacy authentication using Azure Active Directory—This month, we introduced the Azure Active Directory legacy authentication blocking preview, which allows customers to stop legacy authentication attempts as part of their conditional access requirements. Identity attacks most often target overly simple passwords and old authentication protocols. This feature improves the overall security of your IT environment by restricting the ability for clients to use older, less secure authentication technologies.

Enhance data classification across your organization—The new Label Activity Explorer in Office 365 provides a quick overview of how the data in your organization has been labeled—allowing you to investigate risky or abnormal activity. To help you manage labeling across the lifecycle of your organization’s content, we enhanced the Data Governance dashboard with new features like the Data Governance toolbox, added links and tools for common data governance tasks, and provided a single resource for guidance.

Other updates

• Microsoft Teams has reached FedRAMP Moderate Compliance and will start rolling out to U.S. Government Community Cloud (GCC) customers on July 17, 2018.
• Visio Online is now available in Microsoft Teams. Coworkers can now collaborate on Visio Online diagrams from within their team or channel without toggling between apps.
• SharePoint Swoop—our new enterprise reality show—features a team of MVP experts with just three days to help a Microsoft 365 customer modernize their intranet.
• At Computex 2018, we outlined our vision for how partners can build intelligent edge devices and solutions.

The post New to Microsoft 365 in June—streamlining teamwork and security appeared first on Microsoft 365 Blog.

Updating your Microsoft Store listing is the best way to let customers know more about your app or game. You might want to update the description to call out key features, or add new screenshots and video trailers to help your customers see what your product does best. At times, you may also want to update your app’s or game’s price, either on an indefinite basis or as a limited-time sale to entice new customers to make a purchase.

We’ve heard your feedback that you’d like to be able to see these types of changes in the Microsoft Store sooner. We’ve been working hard to improve our publishing times, and we’re pleased to announce that starting today, submissions that make these types of changes without updating packages will be published faster than ever before. After these submissions go through the certification process (which is often completed in a few hours but may take up to three business days), these updates will be reflected in Microsoft Store in less than one hour!

It’s important to note that updated app packages that you submit to Microsoft Store aren’t affected by these improved processing times, so it could still take up to three business days before submissions with updated packages are published in Microsoft Store. (If no packages are updated, the changes should be published within the hour after the certification process is complete.) If you ever experience a delay of longer than three business days for a submission to be published, we encourage you to contact support so that we can investigate.

Note that when publishing a product for the first time, we recommend submitting at least three days before you want your app or game to appear in the Store. For more precise release scheduling, you can use the Schedule section of the Pricing and availability page when you create your submission to specify a precise release date and time (in UTC or in each local market’s time zone, with the ability to use different dates for different markets if desired).

We hope you’ll find the publishing time improvement makes it easier for you to adjust your pricing quickly or get updated listing details in front of your customer base more quickly.

The post Publishing times for Microsoft Store just got faster! appeared first on Windows Developer Blog.

Today’s post was written by Pete Sattler, senior vice president and CIO at John Wiley & Sons.

As we watch the physical book, magazine, and newsprint markets decline by double digits every year, Wiley—with our 210-year heritage of print publishing—is reinventing our culture to adapt to the demands of a new era in the industry. Today, people want to access as much as possible on their mobile devices. We face the challenge of publishing and delivering content solutions in innovative ways to help our customers succeed, no matter where they are in their education or careers. We need to adopt the same modern tools that we advise our customers to deploy for training and retaining top talent. We want to become proficient at working together in the cloud so that we can connect researchers, learners, and professionals with the information, platforms, and tools they need to share scholarly insights on global issues and challenges.

We’re excited about the business opportunities that this new era brings. And to take advantage of these opportunities, we are empowering Wiley employees with a workplace transformation of our own. We’re using Microsoft Office 365 to transition to a new environment that supports a digital-first mindset. For instance, I have completely done away with paper and can work productively from anywhere. It’s all part of a larger shift at Wiley to drive digital transformation in everything we do. By thinking and working in this new atmosphere, we can innovate products that are relevant to how customers want to consume information. At Wiley, we have a culture of learning, and the ability to openly collaborate is important to the ethos of our organization. We chose Office 365 intelligent communication capabilities to further promote that mindset.

The online collaborative workplaces that we are creating with Office 365 reflect the new physical office spaces that Wiley is building. From traditional cubicles to convenient open areas that inspire people to work together, we’re bringing our new culture to life through flexible floorplans and a growing adoption of mobility and virtual teamwork.

Our IT department is co-sponsoring the workplace transformation project with Archana Singh, executive vice president and chief human resources officer. Together, we aim to deliver contemporary tools that employees can use to virtually and seamlessly connect with each other and with our partners and customers. We put together an amazing Office 365 adoption team to design a new intranet that facilitates collaboration and document management, which is essential to our content-heavy business. Dubbed “The Wire,” the intranet is where Wiley employees start their day. It’s a portal that offers all the powerful Office 365 capabilities and keeps everyone up to date on company news.

Wiley employees would likely discover on their own the range of benefits available via The Wire. However, thanks to the tireless efforts of our Office 365 adoption team, we’ve made considerable headway toward adopting a new way of working. The team organized a global product launch and road show with more than 85 classroom sessions in multiple countries. It’s rewarding to see a more connected mindset taking hold at Wiley to help us deliver the products and services customer demand.

—Pete Sattler

For more on Wiley’s digital transformation, read the case study.

The post John Wiley & Sons reinvents its digital culture with the Microsoft Cloud appeared first on Microsoft 365 Blog.

We’re always looking for ways to make developing with Visual Studio faster.  One of the tasks developers do many times a day is launching debugging sessions.  We identified that script debugging added about 1.5s per F5, but only about 15.5% of people actively debugged script using Visual Studio.

Based on the above, in Visual Studio 15.7 we made the decision to turn off script debugging by default to improve the overall experience for most users. If you want to turn the capability back on, you can do it from Tools | Options | Debugging | and check “Enable JavaScript debugging for ASP.NET (Chrome, Edge, and IE):

We also added the following dialog when you attempt to set a breakpoint with script debugging disabled:

When script debugging is ON, Visual Studio automatically stops debugging when the browser window is closed. It will also close the browser window if you stop debugging in Visual Studio. We added the same capability to Visual Studio when script debugging is OFF under Tools | Options | Project and Solutions | Web Projects:

With this option enabled:

• Visual Studio will open a new browser window when debugging starts
• Visual Studio will stop debugging when the browser window is closed

The following matrix shows you all the available options and the expected behavior for each combination:

If you want Visual Studio to return to its default pre-15.7 behavior, all you have to do is enable script debugging in Tools | Options | Debugging | and check “Enable JavaScript debugging for ASP.NET (Chrome, Edge, and IE). If you notice any unexpected behavior with these options please use report a problem in Visual Studio to let us know. If you have any feedback or suggestions regarding this change please let us know on uservoice or simply post a reply to this post.

In Visual Studio 2017 release 15.8 Preview 3 we’re announcing support for Just My Code stepping for C++. In addition to previously supported callstack filtering, the Visual Studio debugger now also supports stepping over non-user-code. As you “Step In”, for example in an algorithm from the Standard library with a custom predicate or in a Win32 API that has a user callback, the debugger will conveniently step into the predicate or callback you provided rather than the library code that will eventually call your code.

After the very warm reception of our debugging improvements in stepping in std::function calls announced last year at CppCon 2017, the team has been working on an universal solution for this debugging challenge that does not require any annotations in library code. 15.8 Preview 3 is available today with this support and we are looking forward to your feedback.

How to enable Just My Code stepping (JMC)

Just My Code stepping for C++ has the following requirements:

• your program is compiled with a new MSVC compiler switch: /JMC. JMC is now on by default for MSBuild projects in all Debug configurations, so just make sure you recompile your project using the latest MSVC compilers in 15.8 Preview 3 or later.
• the PDB for the binary containing the user code is loaded by the debugger, and
• JMC is enabled in Tools > Options > Debugging > General > Enable Just My Code (which is the default).

New “Step In” behavior

When JMC is enabled, the debugger will keep track of which code is user code or system/library code. When stepping into a function that has PDB information, execution will resume until another function that is marked as user code is reached or the current function completes its execution. What this means in practice is that to get to your code, you don’t have to spend your time stepping over countless lines of library code that you’re not interested in or, what happens more often, you can stop juggling giant lists of breakpoints spread all over your codebase.

For example, in the snippet below, without JMC, if you are ambitious enough to “Step In” until you reach the predicate passed as a parameter to the Standard library algorithm, you will have to press F11 (Step In) 140 times! With JMC, it is just one “Step In” command invocation.

STL algorithms

Another example is stepping into Win32 API callbacks. Without JMC, the debugger is not able to tell that some user code will eventually execute, so it steps over the Win32 API call completely without stepping into the user-defined callback. JMC correctly identifies the callback as user code and stops the debugger appropriately.

Win32 API callbacks

Step Into Specific

To explicitly step into a call that may be non-user code, you can take advantage of the “Step Into Specific” command available in the editor’s context menu. This allows you to select a specific function (user code or not) that you want to step into:

Configuring Just My Code for additional 3rd party libraries

The default set of modules and source files that the C++ debugger considers non-user code is encoded in the default.natjmc file under %VSInstallDir%Common7PackagesDebuggerVisualizers and it specifies WinSDK, CRT, STL and ATL/MFC among other things.

You can customize this set of modules and source files by either:

• modifying the central list in %VSInstallDir%Common7PackagesDebuggerVisualizersdefault.natjmc or
• by creating any number of user-specific .natjmc files under the %USERPROFILE%DocumentsVisual Studio 2017Visualizers folder.

For example, to treat all Boost libraries as non-user code, you can create a boost.natjmc in the above folder with the following content.

<?xml version="1.0" encoding="utf-8"?>
<NonUserCode>
 <File Name="*boost*" />
</NonUserCode>

You don’t need to rebuild your user code for these changes to kick in. In the next debugging session, stepping into code that uses Boost will step over the Boost library code and only stop execution if some user code is found on the callstack.

For more details on the .natjmc file format, see the C++ Just My Code documentation page. Note that the .natjmc format also supports marking code as non-user code based on function name, but for stepping performance reasons, we do not recommend using this functionality for functions that are called often or for large group of functions (‘Function’ rules are much slower than ‘Module’ or ‘File’ rules).

3rd party libraries

Under the covers

As mentioned above, JMC functionality is only available for user-code compiled with the new MSVC compiler switch /JMC. This new switch is already on by default for MSBuild projects in Debug configurations. If you’re using a different build system, you will want to make sure you manually add the off-by-default /JMC switch to the debug builds of your project.

/JMC is supported only for binaries that link against the CRT.

To explicitly turn off JMC, you can use the /JMC- switch.

Give us your feedback!

This release is the first Visual Studio 2017 Preview supporting Just My Code stepping. Your feedback is a critical part of ensuring that we can deliver a delightful debugging experience. For any questions, reach out to us via Twitter at @visualc or via email at visualcpp@microsoft.com. For any issues or suggestions, please let us know via Help > Send Feedback > Report a Problem in the IDE.

The animation below, by Shanghai University professor Guy Abel, shows migration within and between regions of the world from 1960 to 2015. The data and the methodology behind the chart is described in this paper. The curved bars around the outside represent the peak migrant flows for each region; globally, migration peaked during the 2005-2010 period and the declined in 2010-2015, the latest data available.

This animated chord chart was created entirely using the R language. The chord plot showing the flows between regions was created using the circlize package; the tweenr package created the smooth transitions between time periods, and the magick package created the animated GIF you see above. You can find a tutorial on making this animation, including the complete R code, at the link below.

Guy Abel: Animated Directional Chord Diagrams (via Cal Carrie)