conceptclang : an implementation model for c++ concepts – an update

1

ConceptClang: An Implementation Model for C++ Concepts – An Update

Larisse VoufoCenter for Research in Extreme Scale Technologies (CREST)Indiana University

Adrian Kothman, Maxwell CrouseCS Undergraduate Mentees

PL-Wonks, November 2013

2

Topics Generic programming with concepts

in C++.

Implementing concepts with ConceptClang for C++,

o in Clang.o in other compilers?

for other languages?o e.g. Chapel?o Talk to Max and Adrian.

3

Topics Generic programming with concepts

in C++.

Implementing concepts with ConceptClang for C++, for other languages?

o e.g. Chapel?

Ongoing theoretical findings: Name binding framework:

o Clarifies complex rules like argument-dependent lookup (ADL) in C++.o Introduces a new scoping rule: weak hiding.o Specifies a new mechanism: two-stage name binding.

Open classes (extensible structures) for free.

4

Outline① Brief Introduction:

1. Generic programming with concepts.2. Constrained C++ templates.

3. Implementing concepts w/ ConceptClang.

② Ongoing theoretical findings:1. Name binding framework.2. Open classes (extensible structures) for free.

③ ConceptClang’ed Chapel, an ongoing undergraduate project. https://github.iu.edu/lvoufo/CCedChapel [branch:mvcrouse]

https://github.iu.edu/lvoufo/CCedChapel


Generic Programming• Different languages support it in various capacities:

Genericity by …– Value: e.g. function abstraction– Type: e.g. (parametric or adhoc)

polymorphism– Function: e.g. functions as values– Structure: e.g. requirements and operations on

types– Property: e.g. properties on type– Stage: e.g. meta-programming– Shape: e.g. datatype-generic

cf. "Datatype Generic Programming”. Gibbons. [In Spring School on Datatype-Generic Programming, volume 4719 of Lecture Notes in Computer Science. Springer-Verlag.]

• C++ supports generic programming via templates.

Generic Programming w/ Concepts• Different languages support it in various capacities:

Genericity by …– Value: e.g. function abstraction– Type: e.g. (parametric or adhoc)

polymorphism– Function: e.g. functions as values– Structure: e.g. requirements and operations on

types– Property: e.g. properties on type– Stage: e.g. meta-programming– Shape: e.g. datatype-generic

cf. "Datatype Generic Programming”. Gibbons. [In Spring School on Datatype-Generic Programming, volume 4719 of Lecture Notes in Computer Science. Springer-Verlag.]

• C++ supports generic programming via constrained templates.

Example: Lifting

int sum(int* array, int n) { int s = 0; for (int i = 0; i < n; ++i) s = s + array[i]; return s;}

template<InputIterator I, typename T, BinaryFunction Op> requires(Assignable<InputIterator::value_type, BinaryFunction<Op>::result_type>)T accumulate(I first, I last, T init, Op bin_op) { for (; first != last; ++first) init = bin_op(init, *first); return init;}

sum(arr,3)

accumulate(arr, arr+3, 0, 1)

Same Complexity.

Example: Requirements Grouping

concept InputIterator<typename X> : Iterator<X>, EqualityComparable<X> {ObjectType value_type = typename X::value_type; MoveConstructible pointer = typename X::pointer; SignedIntegralLike difference_type = typename

X::difference_type;

...

pointer operator->(const X&);};

9

C++ Templates C++ supports generic programming via templates.

Templates alone are not expressive enough, nor safe: Too general. Lengthy and obscure error messages. Encapsulation breakage. Primarily used by experts.

o “Tricks” are even more complex to understand. Worse: semantic errors go undetected.

Example 1: Error Detection and Diagnosis with C++ Templates

vector<void*> v;sort(v.begin(), v.end(), boost::bind(less<int>(),_1,_2));

$ clang++ test.cpp -o example/usr/local/include/boost/bind/bind.hpp:303:16: error: no matching function for call to object of type 'std::less<int>' return unwrapper<F>::unwrap(f, 0)(a[base_type::a1_], a[base_type::a2_]); ^~~~~~~~~~~~~~~~~~~~~~~~~~/usr/local/include/boost/bind/bind_template.hpp:61:27: note: in instantiation of function template specialization 'boost::_bi::list2<boost::arg<1>, boost::arg<2> >::operator()<bool, std::less<int>, boost::_bi::list2<void *&, void *&> >' requested here BOOST_BIND_RETURN l_(type<result_type>(), f_, a, 0); ^/usr/include/c++/4.2.1/bits/stl_algo.h:2501:6: note: in instantiation of function template specialization 'boost::_bi::bind_t<boost::_bi::unspecified, std::less<int>, boost::_bi::list2<boost::arg<1>, boost::arg<2> > >::operator()<void *, void *>' requested here if (__comp(*__i, *__first)) ^/usr/include/c++/4.2.1/bits/stl_algo.h:2591:7: note: in instantiation of function template ...2 errors generated.

Incompatible

Binary Operator!

Example 2: Error Detection and Diagnosis with C++ Templates

vector<int> v;sort(v.begin(), v.end(), not_equal_to<int>());

$ clang++ test.cpp -o example$

(None !?)

Not Valid Ordering!

Example 1: Error Detection and Diagnosis with Constrained Templates

$ clang++ test.cpp -o exampletest.cpp:260:2: error: no matching function for call to 'constrained_sort' constrained_sort(v.begin(), v.end(), boost::bind(less<int>(), _1, _2)); ^~~~~~~~~~~~~~~~./constrained_algo.h:39:6: note: candidate template ignored: constraints check failure [with I = __gnu_cxx::__normal_iterator<void **, std::vector<void *, std::allocator<void *> > >, Cmp = boost::_bi::bind_t<boost::_bi::unspecified, std::less<int>, boost::_bi::list2<boost::arg<1>, boost::arg<2> > >]void constrained_sort(I first, I last, Cmp bin_op) { ^./constrained_algo.h:38:17: note: Concept map requirement not met. Assignable<RandomAccessIterator::value_type, ... ^./constrained_algo.h:37:3: note: Constraints Check Failed: constrained_sort. requires(RandomAccessIterator, StrictWeakOrdering<Cmp>, ^1 error generated.

vector<void*> v;constrained_sort(v.begin(), v.end(), boost::bind(less<int>(),_1,_2));

Example 2: Error Detection and Diagnosis with Constrained Templates

$ clang++ test.cpp -o exampletest.cpp:261:2: error: no matching function for call to 'constrained_sort' constrained_sort(v.begin(), v.end(), not_equal_to<int>()); ^~~~~~~~~~~~~~~~./constrained_algo.h:39:6: note: candidate template ignored: constraints check failure [with I = __gnu_cxx::__normal_iterator<int *, std::vector<int, std::allocator<int> > >, Cmp = std::not_equal_to<int>]void constrained_sort(I first, I last, Cmp bin_op) { ^./constrained_algo.h:37:55: note: Concept map requirement not met. requires(RandomAccessIterator, StrictWeakOrdering<Cmp>, ^./constrained_algo.h:37:3: note: Constraints Check Failed: constrained_sort. requires(RandomAccessIterator, StrictWeakOrdering<Cmp>, ^1 error generated.

vector<int> v;constrained_sort(v.begin(), v.end(), not_equal_to<int>());

PARSING INSTANTIATION

Templates: Compiler Mechanism

Template Definition:

Template Use:

Code Generation:

Specialization:

template<typename Iter, typename T, typename BinOp>T accumulate(…) { … }

vector<int> v;int i = accumulate(v.begin(), v.end(), 0, plus<int>());

vector<int> v;int i = accumulate<vector<int>::iterator, int, plus<int> >(v.begin(), v.end(), 0, plus<int>());

int accumulate(vector<int>::iterator first, vector<int>::iterator last, int init, plus<int> bin_op);

int accumulate(vector<int>::iterator first, vector<int>::iterator last, int init, plus<int> bin_op) { ... }

CheckOnce!

PARSING INSTANTIATION

Constrained Templates: Mechanism

Constrained Template Definition:

Constrained Template Use:

Code Generation:

Specialization + Models:

vector<int> v;int i = accumulate<vector<int>::iterator, int, plus<int> >(v.begin(), v.end(), 0, plus<int>());

int accumulate(vector<int>::iterator first, vector<int>::iterator last, int init, plus<int> bin_op);

int accumulate(vector<int>::iterator first, vector<int>::iterator last, int init, plus<int> bin_op) { ... }

Once!

template<typename I, typename T, typename BinOp>requires(InputIterator, BinaryFunction<Op>, ...)T accumulate(...) { ... }

Check

Constraints-Check

InputIterator<vector<int>::iterator>, BinaryFunction<bin_op>, ...

Concepts: Elementary Components Concept Definition:

• Name + parameters• Requirements• Refinements

• Extends requirements

Constrained Template Definition:• Constraints specification

Concept Model (Template):• Concept id: name + arguments• Requirement satisfactions• Refinement satisfactions

• One for each refinement

Constrained Template Use:• Constraints satisfaction

• ConceptClang extensions affect only 4/~17 components of Clang:

• The Driver component is modified only for compiler flags support.– So, we do not consider it in our analysis.

ConceptClang Infrastructure

Data Structures• AST -- new AST nodes• Basic -- Data structures extensions.

Parse Sema

• ConceptClang extensions affect only 4/~17 components of Clang:

• The Driver component is modified only for compiler flags support.– So, we do not consider it in our analysis.

ConceptClang Infrastructure

Data Structures• AST -- new AST nodes• Basic -- Data structures extensions.

Parse Sema

See C++Now’12 Talk:“ConceptClang: Towards an Implementation Models for C++ Concepts”

19

Preliminary Observation The ConceptClang infrastructure is parameterized by the (type

of) associated requirements. E.g. declarations in PF, expressions in PA, Statements in Chapel…

Main workload is at the infrastructure layer, in Sema: constraints satisfaction, concept model lookup, checking, generating, name rebinding at instantiation time.

So far, only simple name uses, e.g., function calls, are covered. In PF, thus also simple associated declarations and types.

20

Current Limitations In constrained generic components:

Type-checking name uses weak hiding and Two-Stage Name Binding.

Beyond simple function calls:Generalizing name uses ( or Two-Stage Name Binding) structure-opening archetypes Open/Extensible classes/structures for free.

21


1. Generic programming with concepts.2. Constrained C++ templates.3. Implementing concepts w/ ConceptClang.



22

Current Design Limitations No good solution, presently. Our name binding framework explains.

concept Foo2<typename P> = requires (P a, P b) { foo(a, b); }

void foo(int) { }

template<Foo2 T> void gen_func(T a, T b) {

foo(a, b);foo(1);

}

Foo2<T>

::

Should Foo2<T>::foo() shadow ::foo()? How?

Reject or accept the call foo(1)?

23

Our Name Binding Framework Specifies name binding.

void foo();namespace adl { struct X {}; void foo(X); }namespace ns { void foo(int); }void test(adl::X x) {

using ns::foo;foo(x);

}

test

::

ns

adl

Bind * Best viable candidate, or* error

24

Our Name Binding Framework Specifies name binding, independently of the language.



}

test

::

ns

adl

1. Identify and combine scopes.

2. Execute generic name binding.

Bind * Best viable candidate, or* error

25




}

::

ns

adl

testBind * Best viable candidate, or

* error

1. Express scoping rules.

2. Apply scoping rules.

26




}

::

ns

adl


* error

1. Use scope combinators.

2. Identify name binding on an elementary scope (generic).

27




}

::

ns

adl


* error


2. Depends on a Language concept.

28




}

::

ns

adl


* error


2. Instantiate the Language concept.

29

Our Name Binding Framework Abstracts from declarations, references, and scopes.



} Bind * Best viable candidate, or

* error

Scopes as maps of references to sets of matching declarations.

30

Our Name Binding Framework Views name binding as composed of name lookup and resolution.

Name lookup returns the set of matching declarations, for a given reference.



} Bind * Best viable candidate, or

* error

31

1. Expressing Scoping Rules



}

::

ns

adl


* error


2. Instantiate the Language concept.

32

Hiding: Commonly known as “shadowing”.

Merging: Usually the alternative option to “shadowing”.

Opening: [New name] Necessary to describe ADL. A dual of hiding.

Weak Hiding: [New rule] Necessary for (C++) concepts. A sweet middle between

hiding and merging.

The Combinators

33

The Hiding Combinator ( )

void foo();

void test() {

foo(); }

Result: Binds foo() to ::foo().

test

::

34

The Merging Combinator ( )

void foo(); namespace ns { void foo(int);}

void test() { using ns::foo; foo(); }

Result: Finds ns::foo();

Fails to bind foo(). test

::

ns

35

The Weak Hiding Combinator ( )

void foo(); namespace ns { void foo(int);}

void test() { using ns::foo; foo(); }


test

::

ns

36

The Opening Combinator ( )

void foo();

namespace ns { void foo(int); }

namespace adl { struct X {}; void foo(typ); }

void test(adl::X x) { using ns::foo;

foo(x); }

Result: Finds ns::foo();

Enables ADL; Binds foo(x) to adl::foo().

test

::

adl

ns

37

The Opening Combinator ( )

void foo();

namespace ns { void foo(int); }

namespace adl { struct X {}; void foo(typ); }

void test(adl::X x) { using ns::foo; void foo(); foo(x); }

Result: Finds test::foo();

Disables ADL; Fails to bind foo(x).

test

::

adl

ns

38

Applications Understanding current name binding mechanisms:

Argument-dependent lookup (ADL). C++ operators. A cross-language analysis.

Exploring concepts designs Understanding the current limitations. Exploring new solutions:

weak hiding, 2-Stage name binding, and parameterized weak hiding.

Simplifying compiler designs.

39

C++ Operators Example

struct X {}; struct Y {};

void operator+(X, X) { }void operator+(X, Y) { }

void test(X x, Y y) { void operator+(X, X); x + x; x + y; operator+(x, x); operator+(x, y);}

test

::

40

C++ Operators Scoping Ruleswithout ADL

41

C++ Operators Scoping Ruleswith ADL

Empty, since operator is a

reserved keyword.

42


43


44

Problem: Current Limitations Current scoping rules break seemingly valid codes.


void foo(int) { }


foo(a, b);foo(1);

}

Foo2<T>

::

Should Foo2<T>::foo() shadow ::foo()? How?

Reject or accept the call foo(1)?

45

Solution: Weak Hiding


void foo(int) { }


foo(a, b);foo(1);

}

Foo2<T>

::

Foo2<T>::foo() should weakly hide

::foo()!

Accept the call foo(1)!

46

The Weak Hiding Scoping Rule


concept Foo2<typename P> = requires (P a, P b)

{ … }

void foo(int) { }


foo(a, b);foo(1);

}

Concept<T>

::

47

Implementing Weak Hiding

Implementation = Two-Stage Name Binding (Bindx2)1. Bind with inner scope: s1.2. Bind with outer scope: s2.

Bindx2 repeats name binding under different contexts.

48

Bindx2 for C++ Concepts1. Within restricted scope:

up to the outermost restricted scope. Disables ADL and some qualified name lookups.

2. In surrounding scope: normal lookup – including ADL.

concept Foo2<typename P> = requires (P a, P b)

{ … }

void foo(int) { }


foo(a, b);foo(1);

}

Foo2<T>

::

49

Ambiguity for C++ Concepts:What is the most desirable?

1. Ambiguity IS an error, always?

2. Ambiguity IS NOT an error, always?

3. A middle ground option?

Rejects desirable, or binds to undesirable

Accepts undesirable

concept Foo2<typename P> {void foo(P, int);void foo(int, P);

}

template<Foo2 T> void gen_func(T a, int b) {

foo(b, b); foo(b);}

Proposed Extension: Accept only desirable! (?)

50

Ambiguity for C++ Concepts:What is the most desirable?

1. Ambiguity IS an error, always?

2. Ambiguity IS NOT an error, always?

3. A middle ground option?

Rejects desirable, or binds to undesirable

Accepts undesirable

concept Foo2<typename P> {void foo(P, int);void foo(int, P);

}

template<Foo2 T> void gen_func(T a, int b) {

foo(b, b); foo(b);}

Proposed Extension: Accept only desirable! (?)

See C++Now’13 Talk:“Weak Hiding for C++ Concepts and a Generic Way to Understand Name Binding.”

51

C++ Concepts: Proposed Extension

Ambiguity is not an error, when in restricted scope. Ambiguity remains as defined in the language, otherwise.

Parameterize weak hiding over the bind environment.

52

Prototypes Underway

[ https://github.iu.edu/lvoufo/BindIt ]

1. Parser interface for managing scopes: Using Haskell’s Parsec library and a mini-C++ language. Modifying existing compilers: language-c, featherweight Java, etc...

2. Executing Bindx2: Explored in ConceptClang.

3. Alternative compositional view of name binding as composed of some lookup_best and assess.

https://github.iu.edu/lvoufo/BindIt



53

Recap Our name binding framework allows:

expressing the scoping rules of a language, for a given reference, in terms of 3-4 scope combinators, and

reasoning about the application of scoping rules generically, abstracting over the Language concept

incl. the Ambiguity and, optionally, Parameterized concepts.

2-stage name binding (Bindx2): is an implementation of weak hiding, optionally parameterized by the

bind environment, and preserves valid programs in transition from C++ to ConceptC++.

54





55

Current Observations The ConceptClang infrastructure is parameterized by the (type

of) associated requirements. E.g. declarations in PF, expressions in PA, statements in Chapel…

Main workload is at the infrastructure layer, in Sema, incl. weak hiding, 2-stage name binding, and structure-opening archetypes.

Can we prevent compilers from repeating the workload? Just how generic is ConceptClang?

Can we assume that its infrastructure is language-independent as well as design-independent?

56







57

ConceptClang’ed Chapel Reuses the ConceptClang infrastructure to Implement concepts in Chapel independently of the design.

The design can be plugged-in later by instantiating the infrastructure.

Challenges: Different Language:

o From general-purpose C++ to HPC-specific Chapel Different Compiler:

o From modern “by-hand” recursive-descent to Lex/Yacc-generated LALR(1). Somewhat different implementation language:

o From carefully selected subset of C++ to C-like C++.

58

ConceptClang’ed Chapel ConceptClang is design-independent. Is ConceptClang language-dependent?

1. Assume YES:o Parse declarations into ConceptClang’s data structures.o Reuse ConceptClang’s semantic analyzer.

2. Assume NO:o Define a “bridge” interface between compilers and ConceptClang.o The “bridge” shall encapsulate language-dependent implementations.o Compilers provide specialized implementations of the bridge.

3. Merge approaches 1 and 2. Undergraduate Projects:

Max Crouse, Junior: Approach 1 Adrian Kothman, Sophomore: Approach 2

59

ConceptClang Bridge Building

Author: Adrian Kothman

60

The Bridge

• Layer of abstraction between compiler and concepts

61

Purpose

• Ultimate goal is portability• Bridge allows portability

ConceptClang

Clang Compiler X

62

How

Replace concept calls to compiler

Compiler implements bridge

ConceptClang

Clang Compiler X

BridgeCompiler makes calls to ConceptClang through bridge

63

How

Leaves implementation up

to developer

Bridge interface contains needed function declarations

ConceptClang

Clang Compiler X

Bridge

64

How

Remove unneeded indirections

ConceptClang

Clang Compiler X

Bridge

ConceptClang Shell

1. Comment out call to concepts, essentially leaving clang2. Create trivial bridge3. Generalize concept function names

66

Reusing Data Structures

Author: Max Crouse

Generics in Chapel

• What are we focusing on?– Formal Type Arguments

• proc foo(type t) { … }– Formal Arguments Without Types

• proc foo(t) { … }– Formal Arguments with Queried Types

• proc foo(t : ?ty) { … }

A Simple Generic Function

• var A = 5; var B = 7• proc GenEx(ref arg1 : ?t , arg2 : ?tt) {

arg1 = arg1 + arg2;}

• What happens when we call GenEx(A, B)?– A becomes 12

• What about when A and B are strings?

Our Goal• var A = 5; var B = 7;• concept IntConstraint <typename T1, typename T2> {

int operator+(T1 a, T2 b);}

• proc GenEx(ref arg1: ?t, arg2: ?tt) requires IntConstraint<t,tt> {arg1 = arg1 + arg2;

}• What happens when we call GenEx(A, B)?

– A becomes 12• What happens when we pass strings as arguments?

– We get an error message saying that arg1 and arg2 don’t satisfy IntConstraint requirements

Where We AreTrivial Sample Programs -- Delaying Parsing of Contents

INPUT #1:concept A {- some content -} // #1 concept_map A {- some content -} // #2proc test() requires A<T> {

writeln("Hello, Concepts in Chapel... sort of."); }

test(); OUTPUT #1:• Works and prints out “Hello, Concepts in Chapel... Sort of."

INPUT #2:concept A {- some content -} // #1 concept_map B {- some content -} // #2proc test() requires A<T> {

writeln("Hello, Concepts in Chapel... sort of.");

} test(); OUTPUT #2:• error: concept_map B is not associated with globally defined

concept.

INPUT #3:concept A {- some content -} // #1 concept_map A {- some content -} // #2proc test() requires B<T> {


test(); OUTPUT #3:• error: concept B not defined

INPUT #4:concept A {- some content -} // #1 // concept_map A {- some content -} // #2proc test() requires A<T> {


test(); OUTPUT #4:• error: concept_map for A not defined

74

Current State Adrian:

Incrementally defining the bridge, Starting by stripping out all calls to ConceptClang from Clang, and Identifying all calls from ConceptClang to Clang

o E.g. any call starting with S.* or P.*

Max: Delaying parsing of most concepts content until later,

o at CheckParsed() time. Using own user-defined data structures for now. Implement trivial constraints satisfaction.

75

Future Incrementally reuse ConceptClang’s data structures. Reuse ConceptClang’s constraints satisfaction procedure. Incrementally integrate the bridge in. Identify how much of ConceptClang’s parsing can be reused.

Document/report on official website. Submit a paper:

Students in Action Workshop (SiAW) -- Deadline: November 30th. Poster Presentation:

UROC@SoIC -- End of Spring

76

Conclusion Concepts add safety to generic programming.

Current designs of concepts are limited by current scoping rules.

The new weak hiding scoping rule can solve the problem, along with two-stage name binding (Bindx2).

Generalizing function calls (or Bindx2) to all name uses leads to extensible structures for free using ConceptClang’s structure-opening archetypes.

ConceptClang’ed Chapel will guide ConceptClang’s portability, from novice perspectives.

77

Thank You! Questions/Comments? ConceptClang:

[ http://www.crest.iu.edu/projects/conceptcpp ]

Name binding framework:[ https://github.iu.edu/lvoufo/BindIt ]

ConceptClang’ed Chapel: [ https://github.iu.edu/lvoufo/CCedChapel [branch:

mvcrouse] ]

http://www.crest.iu.edu/projects/conceptcpp







conceptclang : an implementation model for c++ concepts – an update

Documents