Taking a tour of Roslyn

It’s a big day for us on the Managed Languages team! As announced at the //BUILD conference earlier today, and as posted by Soma on his blog, we are not just delivering a new preview of Roslyn to all of you, but are in fact moving all of the compiler code to open source! The code will be released and maintained by MS Open Tech, who are our partners in this endeavor.

The goal of open-sourcing the compilers is something that we’ve been working towards for just over a year, and we’re really excited that it’s finally time to make the “big reveal.” (Best of all, we no longer have to try to keep a poker face every time we’re asked about whether or not the compilers will ever be open sourced!) And as this momentous day drew closer, I found myself struggling a bit to write a blog post which could capture the enormity of this moment on behalf of our team. Although Microsoft has actively embraced openness as a part of day-to-day operations for quite some time now, it’s nevertheless very difficult to put into words how exciting this particular change is for us, given how it takes our participation in the world of open source to an entirely new level.

In a vain effort to capture all of that emotion, I inflicted (which is really the only word for it) several drafts on my team and my management, none of which really added anything to the basic facts already planned to be revealed in Soma’s blog. Finally, Jay Schmelzer (our PM director) took me aside and wisely said, “Look, we’ve got your back on the ‘big reveal;’ don’t worry about that. Why don’t you take everyone the rest of the way and show us how it actually works?”

Of course, he was exactly right, and so, here we go…

Installing the preview

Before jumping into the open source code, you should first install the Roslyn preview. To do this, follow these steps:

(1) Browse to the preview site. You’ll be asked to sign in with a Microsoft account (and will have the opportunity to create a free one if you don’t have one already).

(2) Once you are signed in, you’ll be taken to a survey page, where you will just need to fill in your name, preferred e-mail address, and country – this is just so we can stay in touch with folks installing the preview in order to get feedback from them and send new information to them.

(3) After filling out the survey, your Microsoft account will be given permission to access the preview. A “Thank you” page will appear; click the Go to .NET Compiler Platform (“Roslyn”) link that appears on it, as shown here:

(4) The .NET Compiler Platforms (“Roslyn”) main page will be presented to you with a couple of options, as shown here:

Click the “Download” button. This will take you to a standard Connect download details page that describes the download and shows the options for obtaining it. Click the Download button on that page to actually download the preview.

(5) Once downloaded, now you can install it! The team has worked hard to make the installation as absolutely quick and painless as possible. After downloading & running it, you’ll be presented with the following dialog:

Clicking “Install” will cause the older compiler & IDE pieces in Visual Studio to be turned off and the Roslyn versions to be installed and turned on in their place. This takes mere seconds to complete. (The old pieces are not deleted, so if you choose to uninstall the VSIX, you’ll safely revert back to using them.)

(6) Once the installation is finished, you’ll need to restart your instance of Visual Studio (assuming it was open during the installation).

After installing & relaunching Visual Studio, you should now be able to see Roslyn-specific improvements. For example, consider the following reasonably-common code in C#:

static void Main(string[] args)
        {int n1;if (int.TryParse(args[0], out n1))
            {Console.WriteLine(n1);
            }
        }

Using one of the new language features in Roslyn (inline declaration expressions), you can now simplify this to:

static void Main(string[] args)
        {if (int.TryParse(args[0], out var n1))
            {Console.WriteLine(n1);
            }
        }
    }

An example for VB which demonstrates a new feature would be:

Sub Main()Dim s As String = "This is a multiline
            string literal."End Sub

which was impossible using the old VB compiler.

Your older projects should build fine against the new compilers (obviously, speak up if they don’t!), though you should be aware that any changes you make to them which use the new language features won’t work if you uninstall the preview.

Beyond examples of how the language is evolving, the Roslyn preview also showcases many examples of how we are using the new Roslyn compilers to provide stronger IDE functionality. One of my personal favorites is inline renaming: select any symbol or method name, right-click and choose Rename (or just press F2), and start typing a new name right in the program file. Your name changes will be reflected instantly throughout all uses of that name in your solution, all at once, without affecting any other symbols/methods that coincidentally have the same name but in a different context.

In this case, there is no guessing about which names refer to the same symbol; that information is known and exposed, without having to resort to string comparisons bolted on top of the text buffer (as used to be the case), all thanks to the new compilers.

Here’s the fine print: please be aware the set of features that end up in the final version is subject to change before an official version is released – features can and will be added, removed, or modified in the interim, often due to getting feedback from you as to what works and what needs to be improved (see the end of this post for details on how to give feedback). Similarly, the language features showcased in the preview are not locked down, and should be viewed as examples of things that are being considered for the final version. Finally, you may sometimes notice that one language might showcase an IDE or language feature in the preview while the other language doesn’t. You shouldn’t worry about that; although it might be because a given feature simply doesn’t make sense for the other language, it’s far more likely that the developers just haven’t gotten around to it yet, as such things are generally implemented serially from one language to the other. (FWIW, I’m already planning to go into more detail as to how features are vetted and scheduled in a future blog post, in case anyone is curious about that.)

Enlisting in the open source project

Now, if you’d like to take it a step further, you can enlist in the open source project to view and experiment with the code yourself. Here are a couple of important points:

• NuGet is used for packaging, and it’s important to use using version 2.8.1 (or later) of it. If you don’t have that extension (or that version) installed on Visual Studio yet, you can add it by selecting Tools→Extensions and Updates, and then in the Online section, select and install “NuGet Package Manager for Visual Studio 2013.” You can also install it from here, and that’s a great site to visit if you encounter any problems with NuGet.
• Also, you’ll also need the Visual Studio 2013 SDK in order to open the projects – you can download that from here.

Once you’ve got those set up, open Team Explorer in Visual Studio (View→Team Explorer). At top of that window, you’ll see a dropdown labeled “Home” listing whatever your current workspace is:

• In the dropdown menu, select Projects and My Teams→Connect to Team Projects.
• At the bottom of the resulting view, you’ll see a section labeled “Local Git Repositories.” Drop-down the Clone list. You’ll see a dialog similar to this:

• In the first field, type in http://git01.codeplex.com/roslyn (as shown in the picture), and in the second field, specify the location on your local machine where you’d like to store the code.
• Press “Clone.”

You can now open the solution and build it exactly like any other solution. Furthermore, Visual Studio supports the typical Git commands, so that you can (for example) use the Unsynced Commits pane in Team Explorer keep your enlistment up-to-date with any changes that have gone in by using Pull, as shown here:

I encourage you to run Pull often – by keeping your enlistment up-to-date, you’ll be able to see the direction that the compiler work is headed, and to provide feedback on it in our forums.

If you try building Roslyn from an enlistment, please note that Roslyn is built using itself as a requirement, so make sure that you’ve installed the Roslyn preview first before trying that. Don’t try building Roslyn using the old compilers that shipped with VS 2013 – you’ll just get lots of interesting errors which won’t make a lot of sense.

Making local changes in the compilers

When making local changes in the compilers, you’ll want to first create a fork of the code. (The main branch is never modified directly; it only takes pulls coming from forks.) You’ll need to have a CodePlex account to do this first, however. Once you’ve got that, then launch your browser, navigate to https://roslyn.codeplex.com/SourceControl/latest, and click on the “Fork” link:

Give your fork some meaningful name and description, and press “Save”:

You now have your own fork of the code that you can experiment with & update as you see fit without changing your copy of the “main” source. You can clone it locally using the Team Explorer, just as I showed in the previous section, except that you would specify the forked location (e.g., https://git01.codeplex.com/forks/mattgertz/testfork using my example) as the source, and some other appropriate local disk location for where the code would be stored. (Note that you may see a couple of warnings in the build regarding linking to Syntax.xml.Generated.* files. These warnings are benign and will eventually be addressed; you can safely ignore them for now.)

After building your updated compilers, you’ll find the outputs below the Binaries directory off of the root of the enlistment (e.g., in Binaries\Debug, if that’s the configuration you built), and you can use rcsc.exe (for C#) and rvbc.exe (for VB) to compile any test programs that you want to create.

If, at some point in the future, you want to submit one of your changes for consideration in the main source code repository (as opposed to your own fork of it), you’ll be able to send a pull request to the team from inside CodePlex. (Note that the devs are heads down on finalizing this first version of the new compilers, and so during that remaining time, only simple fixes are likely to be accepted in order to avoid churn. You can expect a blog post later on regarding the types of changes that would be accepted, the procedure for verifying stability before submitting, and so on.)

ADVANCED USAGE

It is also possible to update your copy of Visual Studio to use your own built version of Roslyn (for example, to see how the IDE reacts to your changes), but it’s slightly complicated:

First of all, you’ll need to use the release fork, not the master fork. This is because the compiler code is constantly changing in reaction to feedback, and that includes changes to the APIs that are used by the non-open IDE bits in the Roslyn preview in order to access compiler information (until the APIs get locked down as we get closer to completion). When these APIs change, the ability to communicate between the two is lost. The release fork, however, accurately reflects the state of the code at the time that the Roslyn preview was snapped, and so is safe to use as a baseline for this sort of thing. (You can see the fork on the Roslyn CodePlex site by choosing “Source Code” and then opening the “Browsing changes in” dropdown – it’s called “releases\build-preview.”)

To switch to this fork in Git, you will need to execute the following two commands from an appropriate Git prompt in your enlistment:

> Git fetch

> Git checkout – track origin/releases/build-preview

Your git repository will now have the contents of the releases/build-preview branch.  Once you’ve done this, you can switch back and forth between the branches using Git checkout master and git checkout releases/build-preview. (Details on Git usage are beyond the scope of this blog; see http://www.git-scm.com/book/en/Git-Branching-Remote-Branches for more information on branching in Git.)

Second, you’ll need to disable Visual Studio’s strong-name assembly checking for the relevant assemblies first. There’s a script to help with that, which you can find checked into the source code at Src/Tools/Microsoft.CodeAnalysis.Toolset.Open/Scripts/Prepare.bat.

Finally, you need to create/set the environment variable OpenSourceDebug to true in your build environment, so that the code is built with the proper key. In a command line build, you can actually do this while building your changes using msbuild /p:OpenSourceDebug=true. For building inside Visual Studio, you can open a command prompt window, enter set OpenSourceDebug=true, and then launch Visual Studio from there.

With all of that done, make your changes. After building, you’ll find the resultant VSIXs to install under your Binaries directory (e.g., Binaries\Debug).

Please note that we will never accept pull requests for the release fork – we need to keep it pristine and accurately reflecting the state of the code relative to the Roslyn preview bits. Anything you actually want considered for submission would need to be ported to a fork created from the master first.

Example of updating the compiler

For those who attended the //BUILD conference, you may have seen Anders demonstrate one example of a change to the language, in which he added support for French quotes in the language. I’ll go through that example in detail here, so that you can replicate it yourself:

(1) Open your fork of the Roslyn codebase in Visual Studio

(2) The next steps depend on which language you want to alter:

(3) For C# developers:

a. Open the file Src\Compilers\CSharp\Source\Parser\Lexer.cs

b. Scroll down to the method ScanSyntaxToken, andcopy & paste in the code that is highlighted below. This will add the character « (Unicode 0x00AB) as a token which can start a string scan:

            // Start scanning the tokencharacter = TextWindow.PeekChar();switch (character)
            {case '\"':case '\'':case '«':this.ScanStringLiteral(ref info);break;

c. Next, scroll down to the method ScanStringLiteral, and copy & paste the in code that is highlighted below. The character will now become a valid beginning to a string.

private bool ScanStringLiteral(ref TokenInfo info, bool allowEscapes = true)
        {var quoteCharacter = TextWindow.PeekChar();if (quoteCharacter == '\'' || quoteCharacter == '"' || quoteCharacter == '«')
            {
                TextWindow.AdvanceChar();

d. Finally, scroll further down in the same method, and copy & paste the in code that is highlighted below. The » character (U+00BB) will now become a valid terminator to a string, but only if it matches a corresponding earlier « character.

else if (ch == quoteCharacter || (ch == '»' && quoteCharacter == '«'))
                    {
                        TextWindow.AdvanceChar();break;
                    }

e. Build the Roslyn solution. The output will be in the Binaries\Debug directory off of the root of your enlistment.

f. Navigate to the Binaries\Debug directory, and create a file there called Program.cs. Paste the following text into it:

using System.Console;class Program{static void Main(string[] args)
         {
             WriteLine();
             WriteLine(«Welcome to my version of Roslyn!»);
         }
     }

g. Open a command prompt, change directory to the Binaries directory, and enter rcsc.exe Program.cs

h. Now, enter program.exe and see Welcome to my version of Roslyn! successfully print out!

(4) For VB developers, it’s a similar process. However, Visual Basic is trickier because double quotes can also be used in escape sequences for the quotes themselves, which adds a lot of more cases to support. Supporting that would make this example much longer; therefore, humbly begging the reader’s indulgence, I will leave that remainder as an exercise for them. In the meantime…

a. Open the file Src\Compilers\VisualBasic\Source\Scanner\CharacterInfo.vb

b. Scroll down to the IsDoubleQuote function, and change its contents to the highlighted line below (code comments removed for brevity’s sake):

Friend Shared Function IsDoubleQuote(c As Char) As Boolean
            Return c = """"c OrElse (c >= DWCH_LSMART_DQ AndAlso (c = DWCH_DQ Or c = DWCH_LSMART_DQ Or c = DWCH_RSMART_DQ))OrElse c = "«"c OrElse c = "»"c
End Function

c. Build the Roslyn solution. The output will be in the Binaries directory off of the root of your enlistment.

d. Navigate to the Binaries directory, and create a file there called Program.vb. Paste the following text into it:

Imports System.ConsoleModule Module1

    Sub Main()
        WriteLine(vbCrLf &«Welcome to my version of Roslyn!»)
    End Sub

End Module

e. Open a command prompt, change directory to the Binaries directory, and enter rvbc.exe Program.vb

f. Now, enter program.exe and see Welcome to my version of Roslyn! successfully print out!

Saving changes back to the server

Of course, once you’ve finished building and testing your code modifications, you’ll want to push your changes back to server. The typical usages of Git are supported in Visual Studio – code that you’ve modified will be marked with a checkmark, and you can (for example) create a commit (which I tend to think of as a local shelveset, being used to TFS) by right-clicking on a changed file in the solution explorer and choosing Commit, or by using the Changes pane in Team Explorer, as shown below:

Team Explorer also lets you push your changes back to the server at the same time (and bring down any other changes pushed by others) by choosing Commit and Sync, or you can do that separately afterwards from the Unsynced Commits pane in Team Explorer by either doing a full synch with the server, or else pushing individual commits:

Changes that you push will also be reflected on the CodePlex site, and you can also see them in Visual Studio by choosing the View History… action in the Changes pane in Team Explorer:

It’s a brave new world

And so, a new chapter begins for the C# and VB compilers, which will lead to more innovation, more transparency, and the strongest .NET experience for everyone involved. We are really looking forward to partnering together with you in that environment!

As part of that partnership, I want to note that hearing back from you on the preview and the open source code is really important to us, and is what will lead to the best products for you. Furthermore, the (anonymous) telemetry coming back from preview will be of tremendous help as well to gauge the performance and reliability of the code. The Discussions and Issues tabs on the Roslyn CodePlex site are great places to make your voice heard, as well as the comments sections in this blog and the Connect site. The MS Open Tech site is a great place to learn more about their open source investments and to provide feedback about that. Finally, I encourage you look at the Help→Customer Feedback Options dialog in Visual Studio to learn more about on how your telemetry information is used.

There is no doubt that there’s still a lot for us to learn about being part of the open source community, but we’re glad to have all of you along for that ride.

Now, go off and have fun with the preview and the codebase!

--Matt & the Managed Languages Team