Writing a SQLite wrapper component for universal Windows apps

This post was written by Dave Risney, Software Design Engineer in the Operating Systems Group

New to Windows Phone 8.1 is the ability to create and run apps written in JavaScript like you can on Windows 8.1. However, there are some differences in the specific APIs available to apps on Windows Phone 8.1. One such difference is that IndexedDB isn’t available on the Phone and thus presents a challenge for developers writing universal Windows apps in JavaScript that need queryable structured storage. In this post we’ll see how to create a Windows Runtime component through which we can use SQLite, a small, fast, and reliable database library that’s freely available and supported in Windows Store apps. We’ve also provided a working sample app.

Note: There are two existing projects listed below that wrap SQLite in WinRT. You may be able to use these instead of writing your own wrapper. See if they expose the functionality you require and if their license works for you before writing your own. The solution discussed in this post came about specifically to avoid licensing issues.

• SQLite-WinRT on CodePlex: Read more about it in this SQLite-WinRT blog post.
• SQLite3-WinRT on GitHub: Read more about it on their github page.

The Plan

We’ll go through the following steps on the path toward using SQLite in your JavaScript universal Windows app:

1. Start with a Visual Studio solution for an existing JavaScript universal Windows app.
2. Install the SQLite Visual Studio extensions.
3. Create a WinRT component universal apps project.
4. Write the SQLite wrapper code.
5. Write common app code to use the SQLite WinRT wrapper.

We’ll follow this with a look at the implementation of the sample app, a note on converting IndexedDB code to use SQLite, and other related resources.

For the sample app, we started with the IndexedDB samplefor Windows 8.1, made it a universal app, and followed the steps described below.

Install SQLite Visual Studio extensions

The SQLite Development Team has published a SQLite for Windows Runtime (Windows 8.1) Visual Studio extension, making it incredibly easy to add SQLite to your Windows 8.1 Store app solution. Follow the link above, press the Download link, and open the VSIX file to install the extension in Visual Studio.

Similarly, the SQLite Development Team has published a SQLite for Windows Phone 8.1 Visual Studio extension. Follow the same steps to install that extension as well.

Create a WinRT component universal apps project

Although the SQLite extension makes it easy to add a reference and build against SQLite, the SQLite APIs are written in C. To use them from your app, you must wrap the SQLite APIs in a WinRT component that you expose to JavaScript.

Open your app’s solution in Visual Studio and add a new Windows Runtime Component (Universal Apps) project, found under Visual C++ > Store Apps > Universal Apps. This will add Windows, Windows Phone, and Shared projects to your solution for the new WinRT component.

Next, to use the new WinRT component you must add references from the app projects to the corresponding WinRT component projects. Add a reference from the app Windows 8.1 project to the WinRT component Windows 8.1 project, and from the app Windows Phone 8.1 project to the WinRT component Windows Phone 8.1 project.

Now the app can use the WinRT component, but the WinRT component projects still need references to the corresponding SQLite extensions. Add another pair of references, this time from the WinRT component Windows 8.1 project to the SQLite for Windows Runtime (Windows 8.1) extension and from the WinRT component Windows Phone 8.1 project to the SQLite for Windows Runtime (Windows Phone 8.1) extension. The extensions can be found in the Add Reference dialog under Windows (Phone) 8.1, Extensions.

Write the SQLite wrapper code

For details on creating a C++/CX WinRT wrapper, have a look at the Creating Windows Runtime Components in C++ document and Visual C++ Language Reference (C++/CX). Otherwise we’ll describe the minimal WinRT wrapper we created for the sample app. Our goal for the WinRT wrapper is to expose the minimum of what we need in our app and to implement any niceties in JavaScript.

In our case, our app requires opening and closing a database, creating tables, inserting data, and querying data under transactions. The schema required for the sample app is very simple, so our WinRT wrapper contains only a Databaseobject that can open and close a database and execute SQL statements. To make it easy to insert data without worry about escaping, we support binding parameters to my SQL statements. To obtain queried data, we return an array of row objects from our execute method. All of our methods are asynchronous so they don’t block our app’s UI thread while using the database.

In JavaScript we made a few convenience functions to execute code within a transaction and to convert an array of strings containing SQL statements into a JavaScript promise representing asynchronous execution of the SQL statements executed in serial and as a transaction. Because we did this in JavaScript the functions are easy to use with JavaScript functions, promises and arrays. See the Sample app implementation details section below for more specifics on how this was implemented in the sample app.

Your own project may well require wrapping additional SQLite APIs; our example is a simple demonstrative sample that doesn’t take advantage of advanced SQLite features.

Sample app implementation details

Here’s an overview of the various methods in the WinRT SQLite wrapper we’ve made in the sample app.

C++/CX

The SQLite API is written in C and primarily uses UTF-8 char* strings and return values to report errors. WinRT, on the other hand, generally uses UTF-16 Platform::String types and reports errors via exceptions. In the util.* files we implement ValidateSQLiteResult which turns error codes returned from SQLite functions into WinRT exceptions, or passes the return value through for non-error results. Also in util.* are two functions to convert between UTF-8 std::string types we can use with the SQLite APIs and UTF-16 Platform::Stringtypes that we can use with C++/CX and WinRT.

In the Database.* files we implement the Database WinRT class which has only a handful of methods. Here’s the class definition from Database.h:

public ref class Database sealed
{
public:
static Windows::Foundation::IAsyncOperation
            ^OpenDatabaseInFolderAsync(Windows::Storage::StorageFolder ^databaseFolder,
            Platform::String ^databaseFileName);

virtual ~Database();

        Windows::Foundation::IAsyncOperationWithProgress<
            Windows::Foundation::Collections::IVector^,
            ExecuteResultRow^> ^ExecuteAsync(Platform::String ^statementAsString);

        Windows::Foundation::IAsyncOperationWithProgress<
            Windows::Foundation::Collections::IVector^,
            ExecuteResultRow^> ^BindAndExecuteAsync(Platform::String ^statementAsString,
            Windows::Foundation::Collections::IVector
            ^parameterValues);

private:
        Database();

void CloseDatabase();

void EnsureInitializeTemporaryPath();
void OpenPath(const std::string &databasePath);

staticint SQLiteExecCallback(void *context, int columnCount,
char **columnNames, char **columnValues);

        sqlite3 *database;
};

The static OpenDatabaseInFolderAsync method is the only public mechanism to create a Database object. It returns an IAsyncOperation^ meaning it is an asynchronous method and provides the newly created or opened Database object as its asynchronous result. In the implementation we ensure that the SQLite temporary path is configured as described in SQLite documentation, and then we wrap sqlite3_open_v2 using the functions from util.*. We implement the asynchronous operation using PPL create_async. From Database.cpp here’s the definition of OpenDatabaseInFolderAsync:

Windows::Foundation::IAsyncOperation
    ^Database::OpenDatabaseInFolderAsync(Windows::Storage::StorageFolder ^databaseFolder,
    Platform::String ^databaseFileName)
{
return create_async([databaseFolder, databaseFileName]() -> Database^
    {
        Database ^database = ref new Database();
        string databasePath = PlatformStringToUtf8StdString(databaseFolder->Path);
        databasePath += "\\";
        databasePath += PlatformStringToUtf8StdString(databaseFileName);

        database->OpenPath(databasePath);
return database;
    });
}

Database::ExecuteAsync is also asynchronous, this time returning IAsyncOperationWithProgress< IVector^, ExecuteResultRow^>, in which the asynchronous result is a vector of any ExecuteResultRows queried by the executed SQL statement and additionally provides progress notifications containing the same queried rows but provided as soon as they are selected and one at a time. We wrap sqlite3_exec, which is made slightly complicated by sqlite3_exec’s callback parameter mechanism used to provide the queried rows. From Database.cpp, here’s the definition of ExecuteAsync and the callback function provided to sqlite3_exec SQLiteExecCallback:

struct SQLiteExecCallbackContext
{
    Windows::Foundation::Collections::IVector ^rows;
    Concurrency::progress_reporter reporter;
};

Windows::Foundation::IAsyncOperationWithProgress<
    Windows::Foundation::Collections::IVector^,
    ExecuteResultRow^> ^Database::ExecuteAsync(Platform::String ^statementAsString)
{
    sqlite3 *database = this->database;

return create_async([database,
        statementAsString](Concurrency::progress_reporter
        reporter) -> Windows::Foundation::Collections::IVector^
        {
            SQLiteExecCallbackContext context = {ref new Vector(),
                reporter};
            ValidateSQLiteResult(sqlite3_exec(database,
                PlatformStringToUtf8StdString(statementAsString).c_str(),
                Database::SQLiteExecCallback, reinterpret_cast<void*>(&context),
                nullptr));
return context.rows;
        });
}

int Database::SQLiteExecCallback(void *contextAsVoid, int columnCount,
char **columnNames, char **columnValues)
{
    SQLiteExecCallbackContext *context = 
reinterpret_cast(contextAsVoid);
    ExecuteResultRow ^row = ref new ExecuteResultRow(columnCount,
        columnNames, columnValues);

    context->rows->Append(row);
    context->reporter.report(row);

return 0;
}

In order to support SQL parameter binding we also implement Database::BindAndExecuteAsync, which has the same return value as Database::ExecuteAsync but accepts an additional parameter that is a vector of strings that should be bound to the SQL statement. One interesting note is that the IVector^ parameter is tied to the calling thread and so instead we create a copy of the list of strings but as a std::vector. This we capture in our create_async lambda and can use on another thread. Because sqlite3_exec doesn’t support parameter binding we cannot use that convenience function and instead we perform the sqlite3_prepare, sqlite3_bind, sqlite3_step, and sqlite3_finalize sequence explicitly. Here’s the definition of BindAndExecuteAsync from Database.cpp:

Windows::Foundation::IAsyncOperationWithProgress<
    Windows::Foundation::Collections::IVector^,
    ExecuteResultRow^> ^Database::BindAndExecuteAsync(
    Platform::String ^statementAsString,
    Windows::Foundation::Collections::IVector
    ^parameterValuesAsPlatformVector)
{
    sqlite3 *database = this->database;

// Create our own copy of the parameters because the IVector provided
// isn't accessible on other threads.
    std::vector parameterValues;
for (unsignedint index = 0; index < parameterValuesAsPlatformVector->Size; ++index)
    {
        parameterValues.push_back(parameterValuesAsPlatformVector->GetAt(index));
    }

return create_async([database, statementAsString,
        parameterValues](Concurrency::progress_reporter
        reporter) -> Windows::Foundation::Collections::IVector^
    {
        IVector ^results = ref new Vector();
        sqlite3_stmt *statement = nullptr;

        ValidateSQLiteResult(sqlite3_prepare(database,
            PlatformStringToUtf8StdString(statementAsString).c_str(), -1,
&statement, nullptr));

const size_t parameterValuesLength = parameterValues.size();
for (unsignedint parameterValueIndex = 0;
            parameterValueIndex < parameterValuesLength; ++parameterValueIndex)
        {
// Bind parameters are indexed by 1.
            ValidateSQLiteResult(sqlite3_bind_text(statement, parameterValueIndex + 1,
            PlatformStringToUtf8StdString(parameterValues[parameterValueIndex]).c_str(),
            -1, SQLITE_TRANSIENT));
        }

int stepResult = SQLITE_ROW;
while (stepResult != SQLITE_DONE)
        {
            stepResult = ValidateSQLiteResult(sqlite3_step(statement));
if (stepResult == SQLITE_ROW)
            {
constint columnCount = sqlite3_column_count(statement);
                ExecuteResultRow ^currentRow = ref new ExecuteResultRow();

for (int columnIndex = 0; columnIndex < columnCount; ++columnIndex)
                {
                    currentRow->Add(
reinterpret_cast<constchar*>(sqlite3_column_text(statement,
                        columnIndex)), sqlite3_column_name(statement, columnIndex));
                }

                results->Append(currentRow);
                reporter.report(currentRow);
            }
        }

        ValidateSQLiteResult(sqlite3_finalize(statement));

return results;
    });
}

In the ExecuteResultRow.* files we implement ExecuteResultRow and ColumnEntry that contain the results of queries to the database. However these all exist simply to expose this data in a reasonable way to the WinRT caller, and there’s no actual interaction with SQLite APIs here. The most interesting aspect of ExecuteResultRow is merely in how it is used by the Database::*ExecuteAsyncmethods.

JavaScript

In the default.js file we implement a few convenience methods to make it easier to use the WinRT wrapper in our JavaScript app.

The runPromisesInSerial function takes an array of promises and ensures they run one after another to make it easy to run a series of asynchronous ExecuteAsync commands.

function runPromisesInSerial(promiseFunctions) {
return promiseFunctions.reduce(function (promiseChain, nextPromiseFunction) {
return promiseChain.then(nextPromiseFunction);
    },
    WinJS.Promise.wrap());
}

The executeAsTransactionAsync function starts a transaction, runs the function provided by the caller, and then ends the transaction. The only interesting aspect is that the function is asynchronous and to end the transaction we must intercept both asynchronous success and failure of that function, end the transaction, but be sure to still return the same success result or throw the same error value.

function executeAsTransactionAsync(database, workItemAsyncFunction) {
return database.executeAsync("BEGIN TRANSACTION").then(workItemAsyncFunction).then(
function (result) {
var successResult = result;
return database.executeAsync("COMMIT").then(function () {
return successResult;
            });
        },
function (error) {
var errorResult = error;
return database.executeAsync("COMMIT").then(function () {
throw errorResult;
            });
        });
}

The executeStatementsAsTransactionAsync and bindAndExecuteStatementsAsTransactionAsync combine the previous two functions to make it easy to provide an array of strings containing SQL statements that should be executed serially and as a transaction.

function executeStatementsAsTransactionAsync(database, statements) {
var executeStatementPromiseFunctions = statements.map(function statementToPromiseFunction(statement) {
return database.executeAsync.bind(database, statement);
    });

return executeAsTransactionAsync(database, function () {
return runPromisesInSerial(executeStatementPromiseFunctions);
    });
}

function bindAndExecuteStatementsAsTransactionAsync(database, statementsAndParameters) {
var bindAndExecuteStatementPromiseFunctions = statementsAndParameters.map(
function (statementAndParameter) {
return database.bindAndExecuteAsync.bind(database,
                statementAndParameter.statement, statementAndParameter.parameters);
        });

return executeAsTransactionAsync(database, function () {
return runPromisesInSerial(bindAndExecuteStatementPromiseFunctions);
    });
}

Later you can see these functions are used to easily execute a list of SQL statements asynchronously and in serial:

SQLite.Database.openDatabaseInFolderAsync(
    Windows.Storage.ApplicationData.current.roamingFolder, "BookDB.sqlite").then(
function (openedOrCreatedDatabase) {
        database = openedOrCreatedDatabase;
return SdkSample.executeStatementsAsTransactionAsync(database, [
"CREATE TABLE IF NOT EXISTS books (id INTEGER PRIMARY KEY UNIQUE, title TEXT, authorid INTEGER);",
"CREATE TABLE IF NOT EXISTS authors (id INTEGER PRIMARY KEY UNIQUE, name TEXT);",
"CREATE TABLE IF NOT EXISTS checkout (id INTEGER PRIMARY KEY UNIQUE, status INTEGER);"
        ]);
// ...

Moving from IndexedDB to SQLite

It may be the case that you have an existing Windows 8.1 app that uses IndexedDB and you want to make into a universal Windows app. To do this, after completing the steps we describe above, you’ll need to change your app code from using IndexedDB to using the WinRT SQLite wrapper.

Unfortunately there is no easy answer for this situation. For the sample app, we expose a raw SQL contract, use regular SQL tables that require a schema upfront, and represent asynchronous execution with promises. IndexedDB, on the other hand, doesn’t use strict schemas but rather reads and writes JavaScript objects. It is object oriented rather than using SQL statements in strings, and it uses events rather than promises.

Accordingly, the converted code in the sample app looked very different from the original IndexedDB sample app. Generally we were able to maintain the high-level goal of what each database-related method in the sample app did, but it was not a totally straightforward conversion. If you have a lot of existing IndexedDB code you can consider writing your WinRT wrapper so that its interface more closely resembles that of IndexedDB. Hopefully your app’s database code is well separated from the rest of your app or easy to convert.