What’s new for XAML and DirectX interop in Windows 8.1

In Windows 8 we enabled combining the high level UI capabilities of XAML and the low-level rendering power of DirectX within the same app using the three XAML-DirectX interop APIs:

• SwapChainBackgroundPanel, for overlaying XAML UI onto a full-screen DirectX app
• SurfaceImageSource, for drawing DirectX surfaces into a primarily XAML UI
• VirtualSurfaceImageSource, for virtualizing very large-scale DirectX content such as maps

This made XAML the first Windows UI platform to fully integrate both a rich object model and set of controls with high performance DirectX rendering using Direct3D and Direct2D.

We’ve since been pleased to see widespread adoption of this functionality by developers all over the world creating amazing apps ranging from popular games to physics simulations to enterprise apps. Many Microsoft apps – such as Bing Maps, Fresh Paint, OneNote, Reader, and others – also use these approaches to combine XAML UI and custom DirectX graphics:

Example Fresh Paint app structure.

While the SwapChainBackgroundPanel control provided a high-performance rendering solution that made it ideal for DirectX-focused apps like games, it has a number of restrictions: there can be only one active at a time, it always fills the full window, and it has to be at the root of the XAML page layout.

Since the launch of Windows 8, we’ve heard feedback from developers that you wanted the ability to do the same sort of entirely custom 60 FPS DirectX rendering where the app fully controls the swap chain and presentation timing to the screen, but without the layout restrictions imposed by SwapChainBackgroundPanel or the UI thread synchronization required for SurfaceImageSource.

I’m therefore excited to talk about some of the new functionality we’ve introduced in Windows 8.1:

1. The SwapChainPanel control: this control builds upon the functionality of SwapChainBackgroundPanel to provide a great deal more flexibility in rendering, layout, and responding to input.
2. Multi-threaded rendering and Direct2D batching: these new options can help improve your app’s performance when drawing DirectX content to SurfaceImageSource and VirtualSurfaceImageSource surfaces.

Introducing SwapChainPanel

The SwapChainPanel control makes it easier to include 2D and 3D DirectX content within the rest of a XAML app’s UI and render it at 60 FPS: you can now create a DirectX IDXGISwapChain and associate it with a SwapChainPanel control anywhere in the XAML visual tree. The framework then uses DirectComposition to directly compose together all the different types of content you may have in your app, such as DirectX, HTML, media, and XAML.

Similar to the z-order improvements this enabled for WebView, leveraging DirectComposition as a high performance system compositor ensured we could lift many of the restrictions required by the SwapChainBackgroundPanel control:

• A SwapChainPanel can be any size, using standard XAML Height and Width properties
• A SwapChainPanel can be placed anywhere on your XAML page
• A SwapChainPanel can be made transparent using standard DXGI alpha modes
• Multiple SwapChainPanels can be placed on the same page
• XAML rendering properties such as Clip, Opacity, Projection, and RenderTransform work seamlessly across XAML and DirectX content

For example, the following XAML page layout:

Example XAML page layout that includes a transparent SwapChainPanel.

Can now be accomplished using a SwapChainPanel:

<RectangleWidth="200"Height="200"Fill="Orange"/>
<TextBlockText="XAML behind a SwapChainPanel"/>


<SwapChainPanelx:Name="DirectXPanel1"Height="300"Width="300"Opacity="0.5">
<SwapChainPanel.RenderTransform>
<RotateTransformAngle="-25"/>
SwapChainPanel.RenderTransform>

<TextBlockText="XAML as a child element in a SwapChainPanel"/>
SwapChainPanel>

<TextBlockText="XAML overlaid on top of a SwapChainPanel"/>

This improved composability opens up a wider set of scenarios, from interactive 3D visualizations inline on a XAML page to smooth, low latency ink and highlighter rendering on top of text.

Like the other XAML-DirectX interop APIs, you need to query for the underlying ISwapChainPanelNative COM interface so that you can access DirectX types in order to attach a swap chain to the panel:

ComPtr<ISwapChainPanelNative> panelNative;
reinterpret_cast<IUnknown*>(DirectXPanel1)->QueryInterface(IID_PPV_ARGS(&panelNative));
// Associate a swap chain with the SwapChainPanel. 
panelNative->SetSwapChain(m_swapChain.Get());

After calling SetSwapChain() on the UI thread, you can then start rendering and presenting the swap chain. Since the app controls the swap chain presentation timing, you can even do all of your rendering and presentation on a separate background thread.

While the existing SwapChainBackgroundPanel has been maintained to ensure compatibility when upgrading your Windows 8 apps to Windows 8.1, we recommend that apps use SwapChainPanel in the future for drawing swap chain-based DirectX content in XAML apps due to its increased flexibility. Rendering performance is equivalent across both APIs.

SwapChainPanel: Size Synchronization

One of the challenges we faced in designing the SwapChainPanel API was how to enable apps to synchronize size changes across a XAML SwapChainPanel control and its associated DirectX IDXGISwapChain.

One common example is zooming in on a SwapChainPanel inside a ScrollViewer control: if a user pinches to zoom, the system can automatically re-render all of your XAML vector-based content such as text, shapes, and controls at the new size; on the other hand, the app needs to be responsible for re-rendering all content inside a swap chain to ensure it’s crisp and avoid scaling artifacts.

Like all controls, the final rendered size of a SwapChainPanel is determined by the XAML layout and rendering engines to ensure the UI tree is always in a consistent state, but that doesn’t apply to swap chain updates: they intentionally have a very short pipeline through the system in order to minimize latency and ensure apps have full control over rendering and presentation, so there’s no forced synchronization between the XAML control being resized or scaled on the UI thread and the current size of its associated swap chain. The final rendered size may have a large number of factors, including the current monitor’s DPI, the additive scale transforms on the SwapChainPanel or any of its visual parent elements, or a zoom factor applied by one or more ScrollViewers. Therefore, SwapChainPanel provides new CompositionScaleX/Y properties and a CompositionScaleChanged event which apps can use to properly transform the swap chain’s size whenever the render size changes.

Scenario 3 of the SwapChainPanel SDK sample provides an example of how to use the composition scale to resize a swap chain by calling ResizeBuffers() in conjunction with SetMatrixTransform() to synchronize size changes to the same presented frame and thereby prevent scaling artifacts.

SwapChainPanel: Low Latency Input

Another exciting new feature introduced with SwapChainPanel in Windows 8.1 is independent input: you can now opt into receiving user input from mouse, pen, or touch directly on a background MTA thread instead of the app’s ASTA UI thread. The major benefit here is performance: with the ability to both handle input and render directly to a swap chain on a background thread which you create yourself, you now have:

• The lowest possible end-to-end latency between receiving input and rendering in response to it, since input goes directly to your swap chain without being routed through the CoreWindow or the XAML object tree
• A way to keep long-running UI thread operations from blocking your input processing and rendering
• A way to keep your input processing and complex rendering from blocking the UI thread

Moving work off the UI thread can greatly improve application responsiveness: some of our ink rendering tests that averaged 100+ ms responses time saw up to an 80% reduction in end-to-end latency using a SwapChainPanel and independent input!

This noticeably improves the end-user experience and helps you build true stick-to-your-finger experiences for everything from game interaction to drawing highlighted regions in a document using the new MinBlend CompositeMode option:

Highlighting text using the new MinBlend mode.

You opt into independent input by creating a CoreIndependentInputSource object and registering event handlers for any pointer events that are generated when the user interacts with the SwapChainPanel:

// Create a task delegate to register for independent input and begin processing input messages.

auto workItemHandler = ref new WorkItemHandler([this](IAsyncAction ^)
{
// The CoreIndependentInputSource will raise pointer events for the specified device types on whichever thread it's created on.
    m_coreInput = CreateCoreIndependentInputSource(
        CoreInputDeviceTypes::Mouse |
        CoreInputDeviceTypes::Touch |
        CoreInputDeviceTypes::Pen
        );

// Register for pointer events, which will be raised on the background thread.
    m_coreInput->PointerPressed += ref new TypedEventHandler

You can see an example of independent input in action in Scenarios 2 and 4 of the SwapChainPanel SDK sample.

There’s also a default template in Visual Studio 2013 to help you get started: creating a C++ DirectX (XAML) app will automatically set up a SwapChainPanel-based app and register for independent input.

The default Visual Studio template for a SwapChainPanel-based app.

SurfaceImageSource / VirtualSurfaceImageSource Updates

We’ve also made some improvements to SurfaceImageSource and VirtualSurfaceImageSource in Windows 8.1:

• Faster Direct2D drawing to multiple surfaces
• Easier multi-threaded drawing

Both of these are accomplished using the new ISurfaceImageSourceNativeWithD2D native COM interface, which you can query for from an existing SurfaceImageSource or VirtualSurfaceImageSource object.

Faster Direct2D Drawing

Since a SurfaceImageSource or VirtualSurfaceImageSource can be used as a brush to fill XAML elements, we’ve found that people commonly use multiple instances of them on the same page. If these contain only 2D graphics, it’s now possible to improve performance when doing multiple surface updates through the use of Direct2D batching.

Using the new ISurfaceImageSourceNativeWithD2D instead of the old ISurfaceImageSourceNative interface, you can now share a single Direct2D device object across multiple SurfaceImageSources:

// Associate two SurfaceImageSources with the same D2D device to indicate updates should be batched together.

m_sisNative1->SetDevice(m_d2dDevice.Get());
m_sisNative2->SetDevice(m_d2dDevice.Get());

ComPtr d2dContext;

// Begin drawing using a shared D2D device context provided by the system.
m_sisNative1->BeginDraw(updateRect1, IID_ID2D1DeviceContext, &d2dContext, &offset1);



// Draw to first SurfaceImageSource
// ...


// Suspend drawing to the first SurfaceImageSource and begin drawing the second
m_sisNative1->SuspendDraw();
m_sisNative2->BeginDraw(updateRect2, IID_ID2D1DeviceContext, &d2dContext, &offset2);

// Draw to next SurfaceImageSource
// ...

The system will then automatically optimize how the drawing operations for all updates surfaces are sent to the GPU to improve performance.

Multithreaded Rendering

The new ISurfaceImageSourceNativeWithD2D interface also enables you to render to a SurfaceImageSource or VirtualSurfaceImageSource from a background thread. This means you can offload expensive long-running drawing operations to a background thread, which keeps your UI thread unblocked and responsive so the user can continue to interact with the rest of the app.

The highlighted methods can now be freely called from any thread:

interface iISurfaceImageSourceNativeWithD2D : public IUnknown
{
    SetDevice(IUnknown *device);      

BeginDraw(RECT updateRect, REFIID iid, void **updateObject, POINT *offset);

// Suspends drawing on one thread so the current thread so it can be resumed on another.
SuspendDraw();       

ResumeDraw();

    EndDraw();
};

Since updates to a SurfaceImageSource or VirtualSurfaceImageSource are always synchronized with XAML UI thread frames, EndDraw() must still be called on the UI thread to tell the system when to actually update the content displayed on screen. This synchronization is the primary functional difference between a SurfaceImageSource and a SwapChainPanel.

In Closing

We’ve continued to make significant investments in XAML + DirectX interop in Windows 8.1, and hope you find this new functionality useful as you continue to create amazing apps that combine the rich UI and productivity of XAML with the high-performance graphics capability of DirectX.

You can check out the following resources for more info and examples:

• DirectX and XAML interop overview
• SwapChainPanel sample
• XAML DirectX 3D shooting game sample
• The Dev Center Forums

--Jesse Bishop, Senior Program Manager, Windows UI Platform