Expression SFINAE improvements in VS 2015 Update 3

This post written by Xiang Fan and Andrew Pardoe

Last December we blogged about partial Expression SFINAE support in VS 2015 Update 1. Some of the things we heard from you after that post are that you wanted better Expression SFINAE support, especially for popular libraries such as Boost and Range-v3. These libraries have been our focus over the last few months.

What is SFINAE?

As a reminder, SFINAE is an acronym for ‘Substitution Failure Is Not An Error’. The idea is that when the compiler tries to specialize a function template during overload resolution, it is ok if the specialization fails as long as there are other valid candidates. C++11 introduced features like decltype and constexpr, and it is more common to have expressions during the template argument deduction and substitution process. The C++ standards committee clarified the SFINAE rules for expressions in C++11.

Boost support

The big news with this update is that Boost now compiles correctly with MSVC without defining the macro BOOST_NO_SFINAE_EXPR. We worked with the Boost maintainers and community to make sure we got it correct. There are still libraries and open source projects that build on top of Boost that have blocking issues. Please let us know if you run into any issues. You can email us at visualcpp@microsoft.com or use any of the feedback channels below.

There are still three failing tests in the Boost test suite when compiling with MSVC. Two test failures relate to using the noexcept operator in a template type argument. We currently parse it immediately, even in a dependent context. One test failure is a Boost library issue that we’ve reported to the maintainers.

Range-v3 support

While we’ve been testing with Range-v3, our progress isn’t as complete as it is with Boost. We went through all range-v3 tests and identified various issues in it. We logged over 50 bugs internally to track our progress and have fixed about half of these bugs. Issues remain in many different feature areas, such as name lookup, friend functions, variadic templates, alias templates, default template arguments, constexpr and noexcept. There are other issues mentioned in the  What’s Remaining section below.

Improvements since Update 1

Our implementation is still partial , but we’re almost done. We’ve made a lot of progress since Update 1. First, though let’s talk about why it’s so hard to get Expression SFINAE working in the MSVC compiler.

MSVC’s token stream parser

The MSVC compiler has been around since before C++ had templates–we’re now working around design decisions that once made sense. MSVC traditionally took a token stream-based approach to parsing templates. When we encounter a template in your code we capture its body as a sequence of tokens without any attempt to understand what the tokens mean. Storing the body as a stream of tokens makes analysis of trailing return types containing decltype-specifiers imperfect, especially in SFINAE contexts.

We have now implemented a recursive-descent parser that generates high level unbound trees for expressions and employed this to analyze the expression argument of decltype in a much more precise way, allowing a better implementation of Expression SFINAE. The recursive descent parser is a work in progress; currently, it can parse only C++ expressions but we’re going to soon expand it to parse the entire C++ syntax and make it the basis for implementing features such as two-phase name lookup. These features have been almost impossible to implement with the token stream-based parser. As work proceeds, the remaining gaps in Expression SFINAE will also be filled.

If you’d like to read more about the changes we’re making to the parser you can find more in this blog post: Rejuvenating the Microsoft C/C++ Compiler.

What’s working now?

Because we’re now generating parse trees for decltype expressions a number of patterns work correctly in Update 3.

• We’ve implemented checking for dependent expression using the new parse tree in the compiler. That fixes this Connect issue reported for a failure compiling Chromium.
• We’ve implemented ability to distinguish different expressions inside decltype using parse tree. Here’s an example simplified from the Boost thread library:

  template
  struct remove_reference
  {
      typedef T type;
  };
  
  template
  inline T&& forward(typename remove_reference::type& t)
  {
  	return static_cast(t);
  }
  
  template
  inline T&& forward(typename remove_reference::type&& t)
  {
  	return static_cast(t);
  }
  
  template 
  inline auto invoke(Fp && f, A0 && a0, Args && ...args)
  -> decltype((forward(a0).*f)(forward(args)...))
  {
  	return (forward(a0).*f)(forward(args)...);
  }
  
  template 
  inline auto invoke(Fp && f, A0 && a0, Args && ...args)
  -> decltype(((*forward(a0)).*f)(forward(args)...))
  {
  	return ((*forward(a0)).*f)(forward(args)...);
  }

• A couple of test cases simplified from Range-v3 now work.

  int f(int *);
  namespace N {
  	template T val();
  
  	template using void_t = void;
  	template struct trait {};
  	template struct trait()))>> {
  		typedef decltype(f(val())) type;
  	};
  }
  N::trait::type t1;
  struct S {
  	template static T val();
  
  	template using void_t = void;
  
  	template struct trait {};
  	template struct trait()))>> {
  		typedef decltype(f(val())) type;
  	};
  };
  S::trait::type t2;

• Also, this example:

  int g;
  template
  using void_t = void;
  template
  struct S1 {};
  template
  struct S1> {};
  struct S2 {
  	int *g;
  	auto f() -> decltype(S1());
  };

What remains to be done?

• constexpr used inside template arguments:
  Whenever we see a function call in a template non-type argument, we parse and do semantic analysis on it immediately. This will prevent its use in a dependent context. constexpr is commonly used as template non-type argument in range-v3.

  		#include 
  		#if defined(WORKAROUND1)
  		// Alternative for type_trait.
  		template::value> * = nullptr>
  		#elif defined(WORKAROUND2)
  		// If the constexpr is always valid, it can be moved into a helper class
  		template
  		struct helper {
  			static const bool value = T{};
  		};
  		template>::value> * = nullptr>
  		#else
  		// VC does semantic analysis on 'std::is_pointer{}' prematurely
  		// and gives error on the function declaration
  		//   error C2512: 'std::is_pointer<_Ty>': no appropriate default constructor available
  		template{}> * = nullptr>
  		#endif
  		short f(const T &);
  		char f(...);
  		int main()
  		{
  			int *p = nullptr;
  			static_assert(sizeof(f(0)) == sizeof(char), "fail");
  			static_assert(sizeof(f(p)) == sizeof(short), "fail");
                  }

• Decltype and dependent expressions inside alias templates:
  If the definition of alias template contains decltype or dependent expression, we will capture its tokens and re-parse it later. Because the tokens don’t resolve the name (they only store the identifier), sometimes the alias specialization will find the wrong symbol during re-parse and you may get incorrect SFINAE result. For example, these two code snippets:

  // General case
  template struct S {
  	using type = int;
  };
  template using type1 = decltype(S{});
  template using type2 = typename type1::type;
  type2 i;

  // This case impacts SFINAE. Simplified from range-v3
  template struct S1 {};
  
  template T declval();
  
  void f(int *);
  
  #ifdef WORKAROUND
  template 
  using void_t = void;
  template  struct helper {};
  template  struct helper()))>> {
  	typedef decltype(f(declval())) type;
  };
  template
  using S2 = typename helper::type;
  #else
  template
  using S2 = decltype(f(declval()));
  #endif
  
  template> * = nullptr>
  short g(U, T);
  
  char g(...);
  
  void h()
  {
  	 // we are supposed to use 'std::nullptr_t' to specialize S2, 
           // but we incorrectly use 'int' during re-parsing
  	static_assert(sizeof(g(nullptr, 1)) == sizeof(short), "fail");
  	 // we are supposed to use 'int' to specialize S2, 
           // but we incorrectly use 'std::nullptr_t' during re-parsing
  	static_assert(sizeof(g(1, nullptr)) == sizeof(char), "fail");
  }

• Pack expansions used inside decltype expressions:We are currently not able to detect pack expansions inside decltype, so it won’t be expanded correctly in some cases. You may get incorrect SFINAE result because of this.

  // Example from range-v3:
  #ifdef WORKAROUND
  template
  struct helper1 {};
  template
  struct helper2 {
  	// pack expansion works for function template
  	template
  	static short f(helper1()(val()...))> * = 0);
  	template
  	static char f(...);
  
  	static const bool value 
              = sizeof(helper2::f(0)) == sizeof(short);
  };
  
  template
  struct helper {
  };
  template
  struct helper {
  	typedef decltype(val()(val()...)) type;
  };
  
  template
  using result_t = typename helper::value, Fun, Args...>::type;
  #else
  template
  using result_t = decltype(val()(val()...));
  #endif

Send us feedback!

As always, we want to hear from you. If you know of any issues with Expression SFINAE not covered in this post, please send us feedback through Send us your feedback in the comments below, through Connect, or by sending us a mail at visualcpp@microsoft.com. Thank you!