Developers frequently send us questions asking for the best publishing option to test their apps, benefits and limitations of beta publishing, as well as how to test in-app purchase. Testing is an important step in the development process to minimize problems, and this in turn can lead to higher ratings, and this in turn can lead to potentially generate more downloads.
Today I want to share a framework for how to think about the different publishing and testing options, which can help you identify the best alternative for your project. I’ll then share details for testing in-app purchase.
Which options are available?
The Windows Phone Store offers a wide range of publishing options, for different levels of app testing:
1. Run app in the emulator– test app functionality using the emulator
2. Run app on a phone– test app performance using a developer-unlocked phone
3. Publish a beta app– test app with a select group of testers
4. Publish a hidden app– distribute your app to many users, in a production environment
5. Publish a visible app– distribute your app to all users in a production environment, in the public Store
6. Enterprise deployment– distribute an app to employees in your company
Comparing the options
This table summarizes the options, and accompanying benefits and limitations:
Characteristics | ||||||
Best for | Development, functionality testing | Development, performance testing | Testing with selected group of users | Testing the production ready app, confidentiality is not that critical or Microsoft account of users is unknown | Global availability | Employees in company |
Who can use the app? | Developer that has the code | Only one person on that phone | Up to a few thousand users that the developer whitelisted based on Microsoft account email | Any user that has the URL link | All consumers in any of the 191 markets your app is published to | Employees of that company |
Is app available in the public store? | No | No | Yes (can be downloaded only by whitelisted users, and cannot be discovered through search or browse) | Yes (cannot be discovered through search or browse) | Yes | No |
Does the app go through certification? | No | No | No | Yes | Yes | No |
Is the app visible to users or crawlers? | No | No | Partial (metadata could be visible: app name, icon, description, screenshots) | Yes | Yes | No |
Test ads shown through Microsoft Ad SDK? | No (only ‘test ads’ can be shown) | Yes | Yes | Yes | Yes | Yes |
Test in-app purchase using WP API? | Yes (see next section for details) | Yes (see next section for details) | Yes – in-app purchase always free | Yes | Yes | No – in-app purchase is not supported because app is not public |
Key advantages | Rapid code development and testing | Unit testing on a production phone, all features operational, validating app speed and performance | Only whitelisted users can access the full app. More rapid turnaround from submission availability due to no certification validation. | Test in production environment, and take app through certification process, testing large number of anonymous users | Global use | Controlled deployment. Apps can be removed from phone by IT. No time required for public app cert. |
Key disadvantages | Can’t test speed and performance. Can’t test some features easily (multi touch, compass, gyro, ads). | Only one phone at a time, no global testing | Need the Microsoft account emails for every whitelisted users. Need to copy all metadata and In-App purchase again to the production app. | Not use for those who need confidential publishing: users can share the link and crawlers can find the app. Turn-around time is the same as a publically visible app. | Not good for testing as consumers will expect a final product, and will rate the app as a finished app | Enterprise set up requires more time (need key, Symantec cert, central IT process, Company Hub…). Apps only available to employees. |
What is a typical workflow for developing and publishing a new app?
As I develop apps, my typical workflow is the following:
- Test in the emulator– Iterative development and functional testing using the emulators. I test using the multiple resolutions of the Windows Phone 8 emulators, as well as Windows Phone 7.
- Test performance on a phone – Once the core functionality is ready I validate the app performance and memory usage on a variety of phones to ensure the app works well on both Windows Phone 8 and Windows Phone 7 devices with different memory configurations. I run the performance analysis tools to understand, on a real phone, what functions and that are taking more time or using more memory.
- Private beta - Once I finish the development, I sometimes publish a beta app. I do this only for apps that are brand-new, and that explore brand-new concepts that I have not tried before, and where I need feedback. I invite to the beta a selected number of trusted users. This way I can get feedback from a (small) sample of users from a variety of phones. Their feedback always helps me fix important issues before the app is offered to all users.
- Public beta (hidden) - I then publish the app hidden, and validate that everything works correctly, that the app has the right name and I used the correct pricing, and that the trial version is working. I also do thorough testing on in-app purchase.
- Production - Once I finalize all the tests, I publish the app in production, visible, in as many markets as possible.
This example can help you decide what will be your own development/publishing/testing workflow for your apps.
Steps to implement each of the different options
Run project in the emulator
Create your app in Visual Studio, then select the emulator that you want to test on and click the green arrow to run. The emulators available can show how the app will look and run in Windows Phone 7, Windows Phone 8, and on different memory and screen resolution phones. The emulator has a different performance and memory profiling than a real phone, so the emulator is ideal for feature testing, but not good for performance testing. Learn more about this option.
Run project on a phone
Developer-unlock your phone. If you have a Windows Phone 7 device, connect it to your development machine, and run the Zune client. If you are using a Windows Phone 8 device, this step is not needed. Then, run the app “Windows Phone Developer Registration” tool, and unlock your device. If you have too many devices already registered, go to Dev Center, and then Account, and remove one of the devices you are not using.
Connect your developer-unlocked phone, and in Visual Studio, select Device and click the green arrow to run. You can even debug an app on the phone.
Publish a beta app
Compile your app and generate a XAP. The XAP file will be found in your project folder, Bin, Release (or Debug) folder. Then in Dev Center create a new product. During the submission process, select BETA publishing, and then add up to 10,000 Microsoft account emails for beta testing. Learn more about beta publishing. Once the beta app has been published, you’ll need to send the link to your beta testers so they can install the app on their phones.
Publish a hidden app
Compile your app and generate a XAP. The XAP file will be found in your project folder, Bin, Release (or Debug) folder. Then in Dev Center publish your app. During the submission process, select PUBLIC STORE, and turn on “Hide from users browsing or searching the Store”. Be careful not to mark an existing public app as hidden; this would make your existing app no longer appear for users who are searching for, or want to download your app.
Once you finish the publishing process, the app goes through the certification validation. When the app is live, you can use the Link to the Store link, under App properties, to share that link with all the users that you want.
Publish a visible app
This is the standard way to publish your app through Dev Center. Make sure you select as many markets as needed to ensure maximum reach of your app. You can do that in the App info section, and then choose individual markets in Market selection and custom pricing. See how to publish your first app.
Enterprise deployment
Use this option to distribute apps to your employees, without publishing your app in the public store. Read more about the steps and process.
Testing in-app purchase
When you add in-app purchase to your app, it is always important that you test your apps for the most common failure scenarios. As you develop your test plan, there are several different mechanisms that you should understand:
Characteristics | 1. Test using the emulator | 2. Test using a beta app | 3. Test using a production app | 4. Test using the emulator with real in-app purchase |
Best for | Rapid prototyping, testing complex in-app purchase scenarios | Testing by more than one user | Testing production code, with zero or non-zero prices | Rapid testing using the production in-app purchase |
How to enable it | Publish a beta app, and add in-app purchase items to the beta app | Publish your app, hidden, in only one market (to minimize risks). Set in-app purchase prices to $0. Once it is published, test on a real phone. Once testing is finished, update the app, remove the ‘hidden’ flag, set the correct prices, and publish in more markets. | Publish an app, then take the Product ID of the published app and add it to the Visual Studio project. This allows the project to use the production in-app purchase products. | |
Allows resetting purchases | Yes | No | No | Yes |
Requires money to purchase | No | No (always zero price) | Yes (if price is set to > $0) | No (always zero price) |
Testing speed | Very rapid prototyping | Some delay to publish beta app | Delay to wait until app is published | Very rapid prototyping |
Complexity of deployment | High, requires adding mock library | Medium, test like a real app, though requires publishing all the app and in-app purchase again to a production app after testing is finished. | Low, test like a real app | Low, test like a real app |
Risks | Requires changes to the code | Requires publishing app and metadata again | Hidden app could be found by users or crawlers | None |
In my own apps, I usually test using the emulator (1), then publish a hidden app and test using a production app (3), and then continue development using the live in-app purchase (4) once the first version of the app has been published – this lets me test with production apps, and I’m not very concerned about the confidentiality of my apps. Every case is different, so you might find the first two options better because they allow faster prototyping and higher confidentiality in testing.
To make option 4 work, take the product ID of your published app from the app details page in Dev Center, and add it in your Visual Studio project, in the manifest packaging. This way the Visual Studio development product is the same as the ID of the published app.
Testing behavior of an app upgrading from Windows Phone 7.X to 8.0
Another scenario that developers want to test is upgrading an app from Windows Phone 7.x XAP to 8.0 XAP. More developers are adding an 8.0 XAP version of the app to take advantage of new functionality available only in Windows Phone 8 XAPs. The app will have two XAPs: 7.x and 8.0, and Windows Phone 8 devices running the 7.x XAP will be able to upgrade to the 8.0 XAP. It is useful to confirm that your app will function correctly during the upgrade.
To test this behavior, publish a beta app that only has the 7.x XAP. Download and use the app. Then add the 8.0 XAP in Dev Center, and publish the app again. After it is published, the beta app will download as an 'upgrade' in the phone, and you can validate that the upgrade scenario will work correctly before taking your upgrade live.
I hope this framework can help you make the best decision for you when it comes to testing your app, and make sure in-app purchase works as well as possible. We’d love to hear your questions and suggestions for publishing and testing Windows Phone apps. Leave a comment here, or tweet at @wpdev.