Today we released a prototype of a C# feature called “nullable reference types“, which is intended to help you find and fix most of your null-related bugs before they blow up at runtime.
We would love for you to install the prototype and try it out on your code! (Or maybe a copy of it! ![😄]()
) Your feedback is going to help us get the feature exactly right before we officially release it.
Read on for an in-depth discussion of the design and rationale, and scroll to the end for instructions on how to get started!
The billion-dollar mistake
Tony Hoare, one of the absolute giants of computer science and recipient of the Turing Award, invented the null reference! It’s crazy these days to think that something as foundational and ubiquitous was invented, but there it is. Many years later in a talk, Sir Tony actually apologized, calling it his “billion-dollar mistake”:
I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn’t resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.
There’s general agreement that Tony is actually low-balling the cost here. How many null reference exceptions have you gotten over the years? How many of them were in production code that was already tested and shipped? And how much extra effort did it take to verify your code and chase down potential problems to avoid even more of them?
The problem is that null references are so useful. In C#, they are the default value of every reference type. What else would the default value be? What other value would a variable have, until you can decide what else to assign to it? What other value could we pave a freshly allocated array of references over with, until you get around to filling it in?
Also, sometimes null is a sensible value in and of itself. Sometimes you want to represent the fact that, say, a field doesn’t have a value. That it’s ok to pass “nothing” for a parameter. The emphasis is on sometimes, though. And herein lies another part of the problem: Languages like C# don’t let you express whether a null right here is a good idea or not.
Yet!
What can be done?
There are some programming languages, such as F#, that don’t have null references or at least push them to the periphery of the programming experience. One popular approach instead uses option types to express that a value is either None or Some(T) for a given reference type T. Any access to the T value itself is then protected behind a pattern matching operation to see if it is there: The developer is forced, in essence, to “do a null check” before they can get at the value and start dereferencing it.
But that’s not how it works in C#. And here’s the problem: We’re not going to add another kind of nulls to C#. And we’re not going to add another way of checking for those nulls before you access a value. Imagine what a dog’s breakfast that would be! If we are to do something about the problem in C#, it has to be in the context of existing nulls and existing null checks. It has to be in a way that can help you find bugs in existing code without forcing you to rewrite everything.
Step one: expressing intent
The first major problem is that C# does not let you express your intent: is this variable, parameter, field, property, result etc. supposed to be null or not? In other words, is null part of the domain, or is it to be avoided.
We want to add such expressiveness. Either:
- A reference is not supposed to be null. In that case it is alright to dereference it, but you should not assign null to it.
- A reference is welcome to be null. In that case it is alright to assign null to it, but you should not dereference it without first checking that it isn’t currently null.
Reference types today occupy an unfortunate middle ground where both null assignment and unchecked dereferencing are encouraged.
Naively, this suggests that we add two new kinds of reference types: “safely nonnullable” reference types (maybe written string!) and “safely nullable” reference types (maybe written string?) in addition to the current, unhappy reference types.
We’re not going to do that. If that’s how we went about it, you’d only get safe nullable behavior going forward, as you start adding these annotations. Any existing code would benefit not at all. I guess you could push your source code into the future by adding a Roslyn analyzer that would complain at you for every “legacy” reference type string in your code that you haven’t yet added ? or ! to. But that would lead to a sea of warnings until you’re done. And once you are, your code would look like it’s swearing at you, with punctuation? on! every? declaration!
In a certain weird way we want something that’s more intrusive in the beginning (complains about current code) and less intrusive in the long run (requires fewer changes to existing code).
This can be achieved if instead we add only one new “safe” kind of reference type, and then reinterpret existing reference types as being the other “safe” kind. More specifically, we think that the default meaning of unannotated reference types such as string should be non-nullable reference types, for a couple of reasons:
- We believe that it is more common to want a reference not to be null. Nullable reference types would be the rarer kind (though we don’t have good data to tell us by how much), so they are the ones that should require a new annotation.
- The language already has a notion of – and a syntax for – nullable value types. The analogy between the two would make the language addition conceptually easier, and linguistically simpler.
- It seems right that you shouldn’t burden yourself or your consumer with cumbersome null values unless you’ve actively decided that you want them. Nulls, not the absence of them, should be the thing that you explicitly have to opt in to.
Here’s what it looks like:
class Person
{
public string FirstName; // Not null
public string? MiddleName; // May be null
public string LastName; // Not null
}
This class is now able to express the intent that everyone has a first and a last name, but only some people have a middle name.
Thus we get to the reason we call this language feature “nullable reference types”: Those are the ones that get added to the language. The nonnullable ones are already there, at least syntactically.
Step two: enforcing behavior
A consequence of this design choice is that any enforcement will add new warnings or errors to existing code!
That seems like a breaking change, and a really bad idea, until you realize that part of the purpose of this feature is to find bugs in existing code. If it can’t find new problems with old code, then it isn’t worth its salt!
So we want it to complain about your existing code. But not obnoxiously. Here’s how we are going to try to strike that balance:
- All enforcement of null behavior will be in the form of warnings, not errors. As always, you can choose to run with warnings as errors, but that is up to you.
- There’s a compiler switch to turn these new warnings on or off. You’ll only get them when you turn it on, so you can still compile your old code with no change.
- The warnings will recognize existing ways of checking for null, and not force you to change your code where you are already diligently doing so.
- There is no semantic impact of the nullability annotations, other than the warnings. They don’t affect overload resolution or runtime behavior, and generate the same IL output code. They only affect type inference insofar as it passes them through and keeps track of them in order for the right warnings to occur on the other end.
- There is no guaranteed null safety, even if you react to and eliminate all the warnings. There are many holes in the analysis by necessity, and also some by choice.
To that last point: Sometimes a warning is the “correct” thing to do, but would fire all the time on existing code, even when it is actually written in a null safe way. In such cases we will err on the side of convenience, not correctness. We cannot be yielding a “sea of warnings” on existing code: too many people would just turn the warnings back off and never benefit from it.
Once the annotations are in the language, it is possible that folks who want more safety and less convenience can add their own analyzers to juice up the aggresiveness of the warnings. Or maybe we add an “Extreme” mode to the compiler itself for the hardliners.
In light of these design tenets, let’s look at the specific places we will start to yield warnings when the feature is turned on.
Avoiding dereferencing of nulls
First let’s look at how we would deal with the use of the new nullable reference types.
The design goal here is that if you mark some reference types as nullable, but you are already doing a good job of checking them for null before dereferencing, then you shouldn’t get any warnings. This means that the compiler needs to recognize you doing a good job. The way it can do that is through a flow analysis of the consuming code, similar to what it currently does for definite assignment.
More specifically, for certain “tracked variables” it will keep an eye on their “null state” throughout the source code (either “not null” or “may be null“). If an assignment happens, or if a check is made, that can affect the null state in subsequent code. If the variable is dereferenced at a place in the source code where its null state is “may be null“, then a warning is given.
void M(string? ns) // ns is nullable
{
WriteLine(ns.Length); // WARNING: may be null
if (ns != null)
{
WriteLine(ns.Length); // ok, not null here
}
if (ns == null)
{
return; // not null after this
}
WriteLine(ns.Length); // ok, not null here
ns = null; // null again!
WriteLine(ns.Length); // WARNING: may be null
}
In the example you can see how the null state of ns is affected by checks, assignments and control flow.
Which variables should be tracked? Parameters and locals for sure. There can be more of a discussion around fields and properties in “dotted chains” like x.y.z or this.x, or even a field x where the this. is implicit. We think such fields and properties should also be tracked, so that they can be “absolved” when they have been checked for null:
void M(Person p)
{
if (p.MiddleName != null)
{
WriteLine(p.MiddleName.Length); // ok
}
}
This is one of those places where we choose convenience over correctness: there are many ways that p.MiddleName could become null between the check and the dereference. We would be able to track only the most blatant ones:
void M(Person p)
{
if (p.MiddleName != null)
{
p.ResetAllFields(); // can't detect change
WriteLine(p.MiddleName.Length); // ok
p = GetAnotherPerson(); // that's too obvious
WriteLine(p.MiddleName.Length); // WARNING: saw that!
}
}
Those are examples of false negatives: we just don’t realize you are doing something dangerous, changing the state that we are reasoning about.
Despite our best efforts, there will also be false positives: Situations where you know that something is not null, but the compiler cannot figure it out. You get an undeserved warning, and you just want to shut it up.
We’re thinking of adding an operator for that, to say that you know better:
void M(Person p)
{
WriteLine(p.MiddleName.Length); // WARNING: may be null
WriteLine(p.MiddleName!.Length); // ok, you know best!
}
The trailing ! on an expression tells the compiler that, despite what it thinks, it shouldn’t worry about that expression being null.
Avoiding nulls
So far, the warnings were about protecting nulls in nullable references from being dereferenced. The other side of the coin is to avoid having nulls at all in the nonnullable references.
There are a couple of ways null values can come into existence, and most of them are worth warning about, whereas a couple of them would cause another “sea of warnings” that is better to avoid:
- Assigning or passing null to a non-nullable reference type. That is pretty egregious, right? As a general rule we should warn on that (though there are surprising counterarguments to some cases, still under debate).
- Assigning or passing a nullable reference type to a nonnullable one. That’s almost the same as 1, except you don’t know that the value is null – you only suspect it. But that’s good enough for a warning.
- A
default expression of a nonnullable reference type. again, that is similar to 1, and should yield a warning.
- Creating an array with a nonnullable element type, as in
new string[10]. Clearly there are nulls being made here – lots of them! But a warning here would be very harsh. Lots of existing code would need to be changed – a large percentage of the worlds existing array creations! Also, there isn’t a really good work around. This seems like one we should just let go.
- Using the default constructor of a struct that has a field of nonnullable reference type. This one is sneaky, since the default constructor (which zeroes out the struct) can even be implicitly used in many places. Probably better not to warn, or else many existing struct types would be rendered useless.
- Leaving a nonnullable field of a newly constructed object null after construction. This we can do something about! Let’s check to see that every constructor assigns to every field whose type is nonnullable, or else yield a warning.
Here are examples of all of the above:
void M(Person p)
{
p.FirstName = null; // 1 WARNING: it's null
p.LastName = p.MiddleName; // 2 WARNING: may be null
string s = default(string); // 3 WARNING: it's null
string[] a = new string[10]; // 4 ok: too common
}
struct PersonHandle
{
public Person person; // 5 ok: too common
}
class Person
{
public string FirstName; // 6 WARNING: uninitialized
public string? MiddleName;
public string LastName; // 6 WARNING: uninitialized
}
Once again, there will be cases where you know better than the compiler that either a) that thing being assigned isn’t actually null, or b) it is null but it doesn’t actually matter right here. And again you can use the ! operator to tell the compiler who’s boss:
void M(Person p)
{
p.FirstName = null!; // ok, you asked for it!
p.LastName = p.MiddleName!; // ok, you handle it!
}
A day in the life of a null hunter
When you turn the feature on for existing code, everything will be nonnullable by default. That’s probably not a bad default, as we’ve mentioned, but there will likely be places where you should add some ?s.
Luckily, the warnings are going to help you find those places. In the beginning, almost every warning is going to be of the “avoid nulls” kind. All these warnings represent a place where either:
- you are putting a null where it doesn’t belong, and you should fix it – you just found a bug! – or
- the nonnullable variable involved should actually be changed to be nullable, and you should fix that.
Of course as you start adding ? to declarations that should be allowed to be null, you will start seeing a different kind of warnings, where other parts of your existing code are not written to respect that nullable intent, and do not properly check for nulls before dereferencing. That nullable intent was probably always there but was inexpressible in the code before.
So this is a pretty nice story, as long as you are just working with your own source code. The warnings drive quality and confidence through your source base, and when you’re done, your code is in a much better state.
But of course you’ll be depending on libraries. Those libraries are unlikely to add nullable annotations at exactly the same time as you. If they do so before you turn the feature on, then great: once you turn it on you will start getting useful warnings from their annotations as well as from your own.
If they add anotations after you, however, then the situation is more annoying. Before they do, you will “wrongly” interpret some of their inputs and outputs as non-null. You’ll get warnings you didn’t “deserve”, and miss warnings you should have had. You may have to use ! in a few places, because you really do know better.
After the library owners get around to adding ?s to their signatures, updating to their new version may “break” you in the sense that you now get new and different warnings from before – though at least they’ll be the right warnings this time. It’ll be worth fixing them, and you may also remove some of those !s you temporarily added before.
We spent a large amount of time thinking about mechanisms that could lessen the “blow” of this situation. But at the end of the day we think it’s probably not worth it. We base this in part on the experience from TypeScript, which added a similar feature recently. It shows that in practice, those inconveniences are quite manageable, and in no way inhibitive to adoption. They are certainly not worth the weight of a lot of extra “mechanism” to bridge you over in the interim. The right thing to do if an API you use has not added ?s in the right places is to push its owners to get it done, or even contribute the ?s yourself.
Become a null hunter today!
Please install the prototype and try it out in VS!
Go to github.com/dotnet/csharplang/wiki/Nullable-Reference-Types-Preview for instructions on how to install and give feedback, as well as a list of known issues and frequently asked questions.
Like all other C# language features, nullable reference types are being designed in the open here: github.com/dotnet/csharplang.
We look forward to walking the last nullable mile with you, and getting to a well-tuned, gentle and useful null-chasing feature with your help!
Thank you, and happy hunting!
Mads Torgersen, Lead Designer of C#
) Your feedback is going to help us get the feature exactly right before we officially release it.
