Evolving the Reflection API

As many developers have noticed, the reflection APIs changed in the .NET API set for Windows Store apps. Much of .NET’s ability to offer a consistent programming model to so many platforms over the last ten years has been the result of great architectural thinking. The changes to reflection are here to prepare for the challenges over the next decade, again with a focus on great architecture. Richard Lander and Mircea Trofin, program managers respectively from the Common Language Runtime and .NET Core Framework teams, wrote this post. -- Brandon

In this post, we will look at changes to the reflection API for the .NET Framework 4.5 and .NET APIs for Windows Store apps (available in Windows 8). These changes adopt current API patterns, provide a basis for future innovation, and retain compatibility for existing .NET Framework profiles.

Reflecting on a popular .NET Framework API

The .NET Framework is one of the most extensive and productive development platforms available. As .NET developers, you create a broad variety of apps, including Windows Store apps, ASP.NET websites, and desktop apps. A big part of what enables you to be so productive across Microsoft app platforms is the consistency of the developer experience. For many developers, the language – be it C# or Visual Basic – is this common thread. For others, it is the Base Class Libraries (BCL). When you look more closely, you'll see that reflection is the basis for many language, BCL, and Visual Studio features.

Many .NET Framework tools and APIs rely on CLR metadata, accessed via reflection, to operate. Examples include static analysis tools, application containers, extensibility frameworks (like MEF), and serialization engines. The ability to query, invoke, and even mutate types and objects is a focal point for .NET developers in many scenarios.

In the .NET Framework 4, we support a rich but narrow set of views and operations over both static metadata and live objects. As we looked at expanding reflection scenarios, we realized that we first needed to evolve the reflection API to enable further innovation.

We saw the design effort to create .NET APIs for Windows Store apps as the avenue to establish the changes that we needed in reflection. We defined a set of updates to reflection that would enable more flexibility for future innovation. We were able to apply that design to .NET APIs for Windows Store apps and also to the Portable Class Library. At the same time, we were able to compatibly enable that design in the .NET Framework 4.5. As a result, we were able to evolve the reflection API consistently and compatibly across .NET Framework API subsets.

We took an in-depth look at the .NET APIs for Windows Store apps in an earlier .NET Team blog post. In this post, we saw that this new API subset is considerably smaller than the .NET Framework 4.5. We have documented the differences on MSDN. One difference that applies is Reflection.Emit, which is not available to Windows Store apps.

Reflection scenarios

Reflection can be described in terms of three big scenarios:

• Runtime reflection
  • Primary scenario: You can request metadata information about loaded assemblies, types, and objects, instantiate types, and invoke methods
  • Status: Supported
• Reflection-only loading for static analysis
  • Primary scenario: You can request metadata information about types and objects that are described in CLR assemblies, without execution side-effects
  • Status: Limited support
  • Alternatives today: CCI
• Extensibility of reflection
  • Primary scenario: You can augment metadata in either of the two scenarios above
  • Status: Supported, but complicated

Today, we have reasonable support for runtime reflection, but not for the other two scenarios. Given that reflection is such a key API, richer support across all three scenarios would likely enable new classes of tools, component frameworks, and other technologies. Full support is inhibited by some limitations in the reflection API in the .NET Framework 4. Primarily, reflection innovation is constrained by a lack of separation of concepts within the API, particularly as it relates to types. The System.Type class is oriented (per its original design) around the runtime reflection scenario. This is problematic because System.Type is used to represent types across all scenarios. Instead, we would benefit from a broader representation of types, designed to support all three scenarios.

Splitting System.Type into two concepts

System.Type is the primary abstraction and entry point into the reflection model. It is used to describe two related but different concepts, reference and definition, and enables operations across both. This lack of separation of concepts is the primary motivation for changing the reflection API. For example, the following scenarios are either difficult or unsupported with the existing model:

• Reading CLR metadata without execution side-effects
• Loading types from alternate sources other than CLR metadata
• Augmenting type representation (for example, changing shape, adding attributes)

In other parts of the product, we have first-class concepts of reference and definition. At a high level, a reference is a shallow representation of something, whereas a definition provides a rich representation. One needs to look no farther than assemblies, a higher level part of reflection, to see this. The System.Reflection.Assembly class represents assembly definitions, whereas the System.Reflection.AssemblyName class represents assembly references. The former exposes rich functionality, and the latter is just data that helps you get the definition should you want it. That’s exactly the model that we wanted to adopt for System.Type.

In order to achieve a similar split for the System.Type concept and class, we created a new System.Reflection.TypeInfo class and shrunk the meaning of the System.Type class. The TypeInfo class represents type definitions and the Type class represents type references. Given a Type object, you can get the name of the type as a string, without any requirement to load anything more. Alternatively, if you need rich information about a type, you can get a TypeInfo object from a Type object. Given a TypeInfo object, you can perform all the rich behavior that you expect with a type definition, such as getting lists of members, implemented interfaces, or the base type.

The value of Type and TypeInfo

The API changes in the .NET Framework 4.5 were made such that we could evolve the reflection API to deliver new scenarios and value. While we changed the shape of the API, we haven’t yet added the additional features that would deliver the value. This section provides a preview of what that value would look like in practice.

Suppose that we are using a static analysis tool that is implemented with the new reflection model. We are looking for all types in an app that derive from the UIControl class. We want to be able to run this tool on workstations on which the UIControl assembly (which contains the UIControl class) does not exist. In this example, let’s assume that we open an assembly that contains a class that derives from the UIControl class:

class MyClass : UIControl

In the .NET Framework 4 reflection model, the Type object (incorporating both reference and definition) that represents MyClass would create a Type object for the base class, which is UIControl. On machines that don't have the UIControl assembly, the request to construct the UIControl Type object would fail, and so too would the request to create a Type object for MyClass, as a result.

Here you see what the reference/definition split achieves. In the new model, MyClass is a TypeInfo (definition); however, BaseType is a Type (reference), and will contain only the information about UIControl that the (MyClass) assembly contains, without requiring finding its actual definition.

Type baseType = myClassTypeInfo.BaseType;

In other scenarios, you may need to obtain the definition of UIControl. In that case, you can use the extension method on the Type clas, GetTypeInfo, to get a TypeInfo for UIControl:

TypeInfo baseType = myClassTypeInfo.BaseType.GetTypeInfo();

Of course, in this case, the UIControl assembly would need to be available. In this new model, your code (not the reflection API) controls the assembly loading policy.

Once again, in the .NET Framework 4.5, the reflection API still eagerly loads the type definition for the base class. The reflection API implementation is largely oriented around the runtime reflection scenario, which has a bias towards loading base type definitions eagerly. At the point that we build support for the static analysis scenario described earlier, we will be able to deliver on full value of the reference/definition split, made possible by Type and TypeInfo.

Applying the System.Type split to the Base Class Libraries

Changing the meaning of Type and adding TypeInfo made it necessary to ensure consistency in the .NET Framework BCL. The .NET Framework has many APIs that return the Type class. For each API, we needed to decide whether a Type (reference) or a TypeInfo (definition) was appropriate. In practice, these choices were easy, since the API inherently either returned a reference or a definition. You either have access to rich data or you don't. We’ll look at a few examples that demonstrate the trend.

• The Assembly.DefinedTypes property returns TypeInfo.
  • This API gets the types defined in that assembly.
• The Type.BaseType property returns a Type.
  • This API returns a statement of what the base type is, not its shape.
  • The base type could be defined in another assembly, which would require an assembly load.
• The Object.GetType method returns a Type.
  • This API returns a Type, since you only need a representation of a type, not its shape.
  • The type could be defined in another assembly, which would require an assembly load.
  • By returning a Type and not a TypeInfo, we also removed a dependency on the reflection subsystem from the core of the .NET Framework.
• Language keywords, like C# typeof, return a Type.
  • Same rationale and behavior as Object.GetType.

Deeper dive into the reflection model update

So far, we’ve been looking at better abstraction in the reflection API, which is the reference/definition split that we made with Type and TypeInfo. We also made other changes, some of which contributed to the reference/definition split and others that satisfied other goals. Let's dive a little deeper into those changes.

Replacing runtime reflection-oriented APIs

In the .NET Framework 4.5 (and earlier releases), you can call Type.GetMethods() to get a list of methods that are exposed on a given type. Such a list of methods will include inherited methods. Our implementation of the GetMethods method has a particular policy for how it traverses the inheritance chain to get the complete list of methods, including loading assemblies for base types that are located in other assemblies. This approach can sometimes be problematic, since loading assemblies can have side-effects that change the execution of your program. The GetMethods method is an example of the heavy bias that the reflection API has to satisfying runtime reflection scenarios, and therefore, is not appropriate for reflection-only loading scenarios.

For the new model, we introduced the DeclaredMethods property that reports the members that are declared (as opposed to members that are available via inheritance) on a given type. There are several other properties, such as DeclaredMembers and DeclaredEvents that follow the same pattern.

The following example illustrates the difference in the behavior between Type/TypeInfo.GetMethods and TypeInfo.DeclaredMethods, using the .NET Framework 4.5.

class MyClass
{
    public void SomeMethod() { }
}
class Program
{
    static void Main(string[] args)
    {
        var t = typeof(MyClass).GetTypeInfo();
        Console.WriteLine("---all methods---");            
        foreach (MethodInfo m in t.GetMethods()) Console.WriteLine(m.Name);
        Console.WriteLine("=======================");
        Console.WriteLine("---declared methods only---");
        foreach (MethodInfo m in t.DeclaredMethods) Console.WriteLine(m.Name);
        Console.ReadKey();
    }
}    

The output is:

---all methods---
SomeMethod
ToString
Equals
GetHashCode
GetType
=======================
—declared methods only---
SomeMethod

You will notice that the GetMethods method retrieves all the public methods accessible on MyClass – including the ones defined on System.Object, like ToString, Equals, GetType and GetHashCode. DeclaredMethods returns all the declared methods (in this case, one method) on a given type, regardless of visibility, and including static methods.

Adopting current API patterns -- IEnumerable

In the era of the async programming model, the reflection APIs stand out since many of them, such as MemberInfo[], return arrays. As you likely know, arrays need to be fully populated before they are returned from an API. This characteristic is bad for both working-set and responsiveness. In the .NET APIs for Windows Store apps, we have replaced all the array return types with IEnumerable collections. Most of you will appreciate working with this friendlier API pattern, which will likely blend in better with the rest of your code.

We have not yet fully taken advantage of this model yet. In our internal implementation of these APIs, we are still using the arrays that were formerly part of the public API contract. In a later version of the product, we can change the implementation to lazy evaluation without needing an associated change to the public API.

Compatibility across .NET target frameworks

We designed the reflection API updates with a goal of compatibility with existing code. In particular, we wanted developers to be able to share code between the .NET Framework 4.5 and .NET APIs for Windows Store apps.

In .NET APIs for Windows Store apps, TypeInfo inherits from MemberInfo, while Type inherits from Object. Type definitions must inherit from MemberInfo to allow for nested types – types that are members of other types – in the same way that methods, properties, events, fields, or constructors are members of a type. You can see that this inheritance approach makes sense, particularly now that Type is very light-weight.

In the .NET Framework 4.5, TypeInfo inherits from Type, while Type is still a MemberInfo. In order to maintain compatibility with the .NET Framework 4, we could not change the base type of Type. We expect that future .NET Framework releases will maintain this same factoring (that is, Type will continue to be a MemberInfo) for backward compatibility.

However, if you are writing code that targets the .NET Framework 4.5, and you want to use the new reflection model, we encourage you to write that code as a Portable Class Library. Portable Class Library projects that target the .NET Framework 4.5 and .NET APIs for Windows Store apps follow the new model, as described above.

See the figure below for a visual illustration of the reflection type hierarchy in the .NET Framework 4.5 and .NET APIs for Windows Store apps.

Figure: Reflection type hierarchy in the .NET Framework 4.5 and .NET APIs for Windows Store apps

Updating your code to use the new reflection model

Now that you have a fundamental understanding of the new model, let's look at the mechanics of the APIs. Basically, you need to know three things:

1. The Type class exposes basic data about a type.
2. The TypeInfo class exposes all the functionality for a type. It is also a proper superset of Type.
3. The GetTypeInfo extension method enables you to get a TypeInfo object from a Type object.

The following sample code demonstrates the basic mechanics of Type and TypeInfo. It also provides examples of the data that you can get from Type and TypeInfo.

class Class1
{
    public void Type_TypeInfo_Demo()
    {
        //Get a Type
        Type type = typeof(Class1);
        //Gets the name of the type
        String typeName = type.FullName;
        //Gets the assembly-qualified type name
        String aqtn = type.AssemblyQualifiedName;

        //Get TypeInfo via the type
        //Note that .GetTypeInfo is an extension method
        TypeInfo typeInfo = type.GetTypeInfo();
        //Get the list of members
        IEnumerable<members> = typeInfo.DeclaredMembers;
        //You can do many other things with a TypeInfo
    }
}

We have received feedback that this change inserts another step – calling the GetTypeInfo extension method – and that it represents a migration hurdle for developers. This change is opt-in for the .NET Framework 4.5, for compatibility reasons. You do not have to use the GetTypeInfo method or the TypeInfo class if you are targeting the .NET Framework 4.5.

With .NET APIs for Windows Store apps, we had the opportunity to create a fully consistent API, which is why we chose to create a clean split between Type and TypeInfo. As a result, code that targets .NET APIs for Windows Store apps will need to use appropriate combinations of Type and TypeInfo classes and the GetTypeInfo extension method. The same is true for Portable Class Library code that targets both .NET APIs for Windows Store apps and the .NET Framework 4.5.

Writing code for the new reflection API – Windows Store and Portable Class Library

As we discussed above, you'll need to adopt the new reflection model if your code targets .NET APIs for Windows Store apps or you're creating a Portable Class Library project that targets both .NET APIs for Windows Store apps and the .NET Framework 4.5.

For example, you will notice that the Get* methods (for example, GetMethod) described earlier are not available, but are replaced by the Declared* properties (for example, DeclaredMethod). If the Get* methods are not present, the reflection binding constraints (BindingFlags options) are not available either. If you're writing new code, you'll need to follow the new model, and if you're porting code from another project, you'll need to update your code to the same model. We understand that these changes may result in non-trivial migration efforts in some cases; however, we hope that you can see the value that can be achieved with the type/typeinfo split.

While the APIs have changed, you may still need to access inherited APIs and filter results. There are a couple of patterns that you can use to accommodate those changes. We’ll look at those now.

We recommend that you write the code that provides the reflection objects that you need. You’ll actually get a clearer view of what the reflection sub-system does by seeing the code in your source file. Your implementation may also be more efficient than our implementation in the .NET Framework, since we accommodate several uncommon cases.

You'll need some code that is a proxy for GetMethods, but that is implemented in terms of the new reflection API. You might need a replacement for another Get* method, such as GetInterfaces; however, you should find that the GetMethods example equally applies. The most straightforward implementation for GetMethods follows. It walks the inheritance chain of a type, and requests the set of declared methods on each class in that chain.

public static IEnumerable<MethodInfo> GetMethods(this Type someType)
{
    var t = someType;
    while (t != null)
    {
        var ti = t.GetTypeInfo();
        foreach (var m in ti.DeclaredMethods)
            yield return m;
        t = ti.BaseType;
    }
}

Since binding flags are not provided in the new reflection API, you do not immediately have an obvious way to filter results, to public, private, static members, or to choose any of the other options offered by the BindingFlags enum. To accommodate this change, you can write pretty simple LINQ queries to filter on the results of the Declared* APIs, as you see in the following example:

IEnumerable<MethodInfo> methods = typeInfo.DeclaredMethods.Where(m => m.IsPublic);
  

We do offer another pattern as an option for porting code more efficiently. We created the GetRuntimeMethods extension method as a convenience API that provides the same semantics as the existing GetMethods API. Related extension methods have been created as an option for the other Get* methods, such as GetRuntimeProperties, as well. As the API names suggests, they are runtime reflection APIs, which will load all base types, even if they are located in other assemblies. These new extension methods do not support the BindingFlags enum, so the filtering approach suggested with LINQ above also applies.

Both of these suggested patterns are good choices for adopting the new reflection model. Note that if reflection support expands in the future to include reflection-only scenarios for static analysis, GetRuntime* methods would no longer be appropriate, should you want to take advantage of those new scenarios.

Writing code for the new reflection API – .NET Framework 4.5

If your code targets the .NET Framework 4.5, you can opt to use the new model, but you do not have to. The .NET Framework 4.5 API is a superset of old and new reflection models, so all the APIs that you’ve used before are available, plus the new ones.

Portable Class Library projects that target the .NET Framework 4, Silverlight, or Windows Phone 7.5 expose only the old model. In these cases, the new reflection APIs are not available.

Conclusion

In this post, we’ve discussed the improvements that we made to reflection APIs in .NET APIs for Windows Store apps, the .NET Framework 4.5, and Portable Class Library projects. These changes are intended to provide a solid basis for future innovation in reflection, while enabling compatibility for existing code.

For more information, porting guides, and utility extension methods, please see .NET for Windows Store apps overview in the Windows Dev Center.

--Rich and Mircea

Follow or talk to us on twitter -- http://twitter.com/dotnet.