backstage java - making a difference in metaprogrammingzpalmer/publications/bsj... · 2016. 1....

Backstage JavaMaking a Difference in Metaprogramming

Zachary Palmer and Scott F. Smith

The Johns Hopkins University

October 27, 2011

Difference-Based Metaprogramming

• Introduction to Metaprogramming

• Compile-Time Metaprogramming in Java

• Traditional Metaprogramming Model

• Difference-Based Metaprogramming Model

What is metaprogramming?

Metaprogramming• Programs input data and output data.

Input ρ Output

• Metaprograms input programs (or programfragments) and output the same.

ρ φ ρ′

Metaprogramming• Programs input data and output data.

Input ρ Output

• Metaprograms input programs (or programfragments) and output the same.

ρ φ ρ′

Examples of Metaprogramming

• C Macros

• C++ Templates

• LISP Macros

• Template Haskell

• MetaOCaml

• Stratego

• Groovy

• etc. etc.

Classifying Metaprogramming

• When is it run?• Compile-time (static)• Runtime (dynamic)

• How are programs represented?• Textually (strings)• Lexically (tokens)• Structurally (ASTs)• Semantically (various structures)

• Which language is used to metaprogram?• Same language (homogenous)• Different language (heterogeneous)

from Accomplishments and Research Challenges in Metaprogramming (Tim Sheard)

Classifying Metaprogramming

• When is it run?• Compile-time (static)• Runtime (dynamic)

• How are programs represented?• Textually (strings)• Lexically (tokens)• Structurally (ASTs)• Semantically (various structures)

• Which language is used to metaprogram?• Same language (homogenous)• Different language (heterogeneous)

from Accomplishments and Research Challenges in Metaprogramming (Tim Sheard)

Template Haskell Example

let mkExp n v =

if n == 0

then [| 1 |]

else [| $(v) * $(mkExp (n-1) v) |]

let f n =

let funNm = mkName ("exp" ++ (show n)) in

let params = [varP (mkName "x")] in

funD funNm $ [clause params

(normalB $ mkExp n (varE $ mkName "x")) []]

mapM f [2..50]

Template Haskell Example

exp2 x = x * x

exp3 x = x * x * x

exp4 x = x * x * x * x

exp5 x = x * x * x * x * x

exp6 x = x * x * x * x * x * x...

......

exp50 x =

50︷︸︸︷x * x * . . . * x * x

Template Haskell

• Programmatic code generation

• Literal syntax for AST construction

• Functional programming style

• Very limited ability to inspect environment

Template Haskell

Why not Template Java?

Template Java?public class Location {

private int x;

public int getX() { return this.x; }

public void setX(int x) { this.x = x; }

private int y;

public int getY() { return this.y; }

public void setY(int y) { this.y = y; }

public Location(int x, int y) {

this.x = x;

this.y = y;

public String toString() {

return "("+this.x+","+this.y+")";

private int x;

public int getX() { return this.x; }

public void setX(int x) { this.x = x; }

private int y;

this.x = x;

this.y = y;

$( property( [|private int x|] )

private int y;

this.x = x;

this.y = y;

private int y;

this.x = x;

this.y = y;

$( property( [|private int y|] )

this.x = x;

this.y = y;

this.x = x;

this.y = y;

$( makePropertyConstructor(

[|int x|], [|int y|]) )

• Metaprograms can’t react to surrounding code• Metaprogrammers compensate by duplicating

information• Functional metaprogramming in a declarative

object-oriented language

[|int x|], [|int y|]) )

• Metaprograms can’t react to surrounding code

• Metaprogrammers compensate by duplicatinginformation

• Functional metaprogramming in a declarativeobject-oriented language

[|int x|], [|int y|]) )

information

• Functional metaprogramming in a declarativeobject-oriented language

[|int x|], [|int y|]) )

information• Functional metaprogramming in a declarative

object-oriented language

What Do We Want?

• Object-oriented, declarative metaprogrammingstyle

• Awareness of surrounding code

• Modular, independent metaprograms

What Do We Want?

Backstage Java

How about some of this?

@@GenerateConstructorFromProperties

public class Location {

@@Property private int x;

@@Property private int y;

Harder than it looks...

Backstage Java

How about some of this?

Harder than it looks...

Traditional Metaprogramming

• Metaprograms are a series of programtransformations

• Each available transformation occurs exactlyonce

• True even for embedded syntax

Embedded Metaprogram Semantics

φ1 φ2

How do we pick?

φ1 φ2

How do we pick?

φ1 φ2

How do we pick?

φ1 φ2

How do we pick?

φ1 φ2

How do we pick?

φ1 φ2

How do we pick?

Ambiguity in Metaprogramming

Assuming three transformations φ1, φ2, φ3...

? ? ? ? ? ?

OpenJava, Groovy

Ambiguity in Metaprogramming

Assuming three transformations φ1, φ2, φ3...

? ? ? ? ? ?

OpenJava, Groovy

Total Ordering Solution

Declaring φ2 before φ1, which is before φ3.

Total Ordering Solution

Declaring φ2 before φ1, which is before φ3.

Necessary Commutation Solution

Requiring that φi ◦ φj = φj ◦ φi for all i and j

Template Haskell, MetaOCaml, LISP, ...

Necessary Commutation Solution

Requiring that φi ◦ φj = φj ◦ φi for all i and j

Template Haskell, MetaOCaml, LISP, ...

Hybrid Solution

Suppose that φ1 and φ2 commute.Suppose that φ3 must occur after them.

Backstage Java*

Hybrid Solution

Suppose that φ1 and φ2 commute.

Suppose that φ3 must occur after them.

Backstage Java*

Hybrid Solution

Suppose that φ1 and φ2 commute.

Suppose that φ3 must occur after them.

Backstage Java*

Hybrid Solution

Backstage Java*

Hybrid Solution

Backstage Java*

Hybrid Solution

Backstage Java*

Hybrid Solution

Backstage Java*

Hybrid Solution

Backstage Java*

Commuting Transformations

How do we tell if φ1 and φ2 commute?

Determining whether or not two arbitrarytransformations commute is undecidable!

Difference-Based MetaprogrammingTreat metaprograms as transformation generators:

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

• Language of δ̄ is not Turing-complete

• Each δ̄ is generated on a case-by-case basis

• No practically significant loss of expressiveness

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

δ̄ can express:

• Creation of a node

• Assignment to a node property

• Additions before or after an element in a list

• Removal of an element from a list

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

δ̄ can express:

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

δ̄ can express:

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

δ̄ can express:

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

Now, prove commutation over pairs of Jδ̄K.

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

φ(ρ) = δ̄

Jδ̄K(ρ) = ρ′

Now, prove commutation over pairs of Jδ̄K.

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

A Simple BSJ Examplepublic class Example {

public static void main(String[] arg) {

BlockStatementListNode list = context.getAnchor().

getNearestAncestorOfType(

BlockStatementListNode.class);

list.addFirst(

<:System.out.println("Hello, world!");:>);

list.addLast(

<:System.out.println("How are you?");:>);

A Simple Examplepublic class Example {

System.out.println("Hello, world!");

System.out.println("How are you?");

• Replace metaprograms with anchors

• Run each metaprogram to collect its changes

• φ1(ρ) = δ̄1 (“Hello, world!” first)• φ2(ρ) = δ̄2 (“How are you?” last)

• Prove that δ̄1 and δ̄2 commute.

• Execute δ̄1 and δ̄2 in some order.

• Replace metaprograms with anchors• Run each metaprogram to collect its changes

• φ1(ρ) = δ̄1 (“Hello, world!” first)

• φ2(ρ) = δ̄2 (“How are you?” last)

An Example of Conflictpublic class Example {

list.addFirst(

list. addFirst (

• φ1(ρ) = δ̄1 (“Hello, world!” first)• φ2(ρ) = δ̄2 (“How are you?” first)

• Now, δ̄1 and δ̄2 do not commute!

• But we can detect this!

Conflict DetectionRecord Node Creation Ruleη 7→ v̂ /∈ ρ ρJη 7→ {l 7→ v̂}K⇒ ρ

(+R η(l = v̂)) ρ⇒ ρ′

List Node Creation Ruleη 7→ v̂ /∈ ρ

ρJη 7→ [(.,M, ∅), (/,M, ∅)]K⇒ ρ′

(M � +Lη) ρ⇒ ρ′

Record Assignment Ruleη 7→ R ∈ ρ ρJη 7→ RJl 7→ v̂KK⇒ ρ

(η.l ← v̂) ρ⇒ ρ′

List Add Before RuleMη3 6= . η1 7→ L ∈ ρ

◦η3 = Σ(Mη3,M,L) L = [◦η′, ◦η3,

◦η′′]

L′ = [◦η′, (η2,M, ∅), ◦η3,◦η′′]

ρJη1 7→ L′K⇒ ρ

(M � η1 : η2 " Mη3) ρ⇒ ρ

List Add After RuleMη3 6= / η1 7→ L ∈ ρ

◦η3 = Σ(Mη3,M,L) L = [◦η′, ◦η3,

◦η′′]

L′ = [◦η′, ◦η3, (η2,M, ∅), ◦η′′]

(M � η1 :Mη3 # η2) ρ⇒ ρ

List Remove Ruleη1 7→ L ∈ ρ◦η2 = (η2,M

′,S) = Σ(η2,M,L)

L = [◦η′, ◦η2,◦η′′]

L′ = [◦η′, (η2,M′,S ∪ {M}), ◦η′′]

(M � η1 : ↓ η2) ρ⇒ ρ′

Recursive Application Ruleδ′ e ⇒ ρ δ ρ⇒ ρ

δ (δ′ e)⇒ ρ′

Value Rule

v̂ ⇒ v̂

Record Assignment Conflict Rulev̂ 6= v̂′

η.l ← v̂ = η.l ← v̂′

Add Before Conflict Ruleω(η2) ω(η′2)

η1 : η2 " Mη3 = η1 : η′2 " M

Add After Conflict Ruleω(η2) ω(η′2)

η1 :Mη3 # η2 = η1 :

Mη3 # η

Unordered Creation Conflict Ruleδ = +R η(l 7→ v̂) ∨ δ = +Lη η ∈ δ′

δ = δ′

Conflict DetectionRecord Node Creation Ruleη 7→ v̂ /∈ ρ ρJη 7→ {l 7→ v̂}K⇒ ρ

(+R η(l = v̂)) ρ⇒ ρ′

List Node Creation Ruleη 7→ v̂ /∈ ρ

ρJη 7→ [(.,M, ∅), (/,M, ∅)]K⇒ ρ′

(M � +Lη) ρ⇒ ρ′

Record Assignment Ruleη 7→ R ∈ ρ ρJη 7→ RJl 7→ v̂KK⇒ ρ

(η.l ← v̂) ρ⇒ ρ′

List Add Before RuleMη3 6= . η1 7→ L ∈ ρ

◦η3 = Σ(Mη3,M,L) L = [◦η′, ◦η3,

◦η′′]

L′ = [◦η′, (η2,M, ∅), ◦η3,◦η′′]

(M � η1 : η2 " Mη3) ρ⇒ ρ

List Add After RuleMη3 6= / η1 7→ L ∈ ρ

◦η3 = Σ(Mη3,M,L) L = [◦η′, ◦η3,

◦η′′]

L′ = [◦η′, ◦η3, (η2,M, ∅), ◦η′′]

(M � η1 :Mη3 # η2) ρ⇒ ρ

List Remove Ruleη1 7→ L ∈ ρ◦η2 = (η2,M

′,S) = Σ(η2,M,L)

L = [◦η′, ◦η2,◦η′′]

L′ = [◦η′, (η2,M′,S ∪ {M}), ◦η′′]

(M � η1 : ↓ η2) ρ⇒ ρ′

Recursive Application Ruleδ′ e ⇒ ρ δ ρ⇒ ρ

δ (δ′ e)⇒ ρ′

Value Rule

v̂ ⇒ v̂

Record Assignment Conflict Rulev̂ 6= v̂′

η.l ← v̂ = η.l ← v̂′

Add Before Conflict Ruleω(η2) ω(η′2)

η1 : η2 " Mη3 = η1 : η′2 " M

Add After Conflict Ruleω(η2) ω(η′2)

η1 :Mη3 # η2 = η1 :

Mη3 # η

Unordered Creation Conflict Ruleδ = +R η(l 7→ v̂) ∨ δ = +Lη η ∈ δ′

δ = δ′

Huzzah!

• Metaprogram conflicts are detected at compiletime

• Metaprograms are still aware of theirsurroundings

Huzzah!

Dependencies

So how do we resolve the conflict?

Dependenciespublic static void main(String[] arg) {

#depends foo; ← Declare target dependency

list.addFirst(

#target foo; ← Declare target membership

list.addFirst(

#depends foo; ← Declare target dependency

list.addFirst(

#target foo; ← Declare target membershipBlockStatementListNode list = context.getAnchor().

list.addFirst(

#depends foo; ← Declare target dependencyBlockStatementListNode list = context.getAnchor().

list.addFirst(

#target foo;

← Declare target membership

list.addFirst(

#depends foo;

← Declare target dependency

list.addFirst(

#target foo;

← Declare target membership

list.addFirst(

Dependency Graph

• One node per metaprogram

• M2 is a member of target “foo”

• M1 depends on the target “foo”

• Therefore, M1 depends on M2 (M1 M2)

• No more requirement for them to commute!

Dependency Graph

M1 M2foo

Dependency Graph

M1 M2foo

Dependency Graph

M1 M2foo

Dependency Graph

M1 M2foo

Dependency Graph

M1 M2foo

Dependency Graph

M1 M2foo

Dependency Graph

M1 a M4 c M6

M2 M3 b M5 M7

/* M1 */ [: #target a; :]

/* M2 */ [: #target a; :]

/* M3 */ [: #target a, b; :]

/* M4 */ [: #target c; #depends a; :]

/* M5 */ [: #target c; #depends b; :]

/* M6 */ [: #depends c; :]

/* M7 */ [: :]

Dependency Graph

M1 a M4 c M6

M2 M3 b M5 M7

• M6 depends on M2 - no obligation to commute

• No path between M5 and M4 - must commute

• No path to M7 - must always commute

• More paths means less obligation to provecommutativity

Dependency Graph

M1 a M4 c M6

M2 M3 b M5 M7

Dependency Graph

M1 a M4 c M6

M2 M3 b M5 M7

Dependency Graph

M1 a M4 c M6

M2 M3 b M5 M7

A more practical example...

...but first, a new feature

A more practical example...

...but first, a new feature

Meta-Annotations

• Declarative metaprogramming abstraction

• Specifies metaprogram code and dependencies

• Can annotate any declaration or blockstatement

• Allows easy reuse of metaprogrammingconstructs

• Here defined by user class named Property

Meta-Annotations

Meta-Annotation Dependencies@@GenerateConstructorFromProperties

• Meta-annotation defns. include dependencies• @@Property is a member of target “property”• @@GenerateConstructorFromProperties

depends on “property”

• Meta-annotation defns. include dependencies

• @@Property is a member of target “property”• @@GenerateConstructorFromProperties

• Meta-annotation defns. include dependencies• @@Property is a member of target “property”

• @@GenerateConstructorFromProperties

• Meta-annotation defns. include dependencies• @@Property is a member of target “property”• @@GenerateConstructorFromProperties

BSJ Dependency Graph

@@Property @@Property

property

• @@Property participates in “property” target

property

• φ3 depends on φ1

property

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

• Execute φ1 obtaining δ̄1.

• Prove Jδ̄1K and Jδ̄2K commute.

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

φ2 +getY

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

• Apply Jδ̄1K and Jδ̄2K to ρ.

• Execute φ3 on the result to get δ̄3.

• Apply Jδ̄3K to get the final object program.

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Execution Example

Jδ̄1K Jδ̄2K

Jδ̄2K Jδ̄1K

Jδ̄3K

Difference-Based MetaprogrammingSummary

• Ambiguities detected at compile-time

• Metaprograms can inspect their environments

• Modular, declarative metaprogramming style

• Suitable for OO languages like Java

Backstage Java Implementation

• Working reference implementation available

• Includes source (∼50k SLOC)

• Full superset of Java 1.6

BSJ Standard Library – @@Memoized

@@Memoized

• Class for generating images

• Each image is generated from pair of Colors

• Memoizing image generation routine forperformance

• Store cached images in a private Map keyed byinput to generation method

@@Memoized

public class ImageGenerator {

@@Memoized

public Image gen(Color a, Color b) {· · · }}

@@Memoized

private static class Key {

private Color a;

private Color b;

· · ·}

public Image gen(Color a, Color b) {· · · }

public Image gen(Color a, Color b) {

/* return cache value, igen as needed */

private Color a;

private Color b;

· · ·}

private Map<???,Image> cache = new · · ·public Image gen(Color a, Color b) {· · · }

private Color a;

private Color b;

· · ·}

private Map<???,Image> cache = new · · ·public Image gen(Color a, Color b) {· · · }

private Color a;

private Color b;

· · ·}

private Map<Key,Image> cache = new · · ·public Image gen(Color a, Color b) {· · · }

private Color a;

private Color b;

· · ·}

private Map<Key,Image> cache = new · · ·private Image igen(Color a, Color b) {· · · }

private Color a;

private Color b;

· · ·}

private Map<Key,Image> cache = new · · ·private Image igen(Color a, Color b) {· · · }public Image gen(Color a, Color b) {

@@Memoized

Questions?

Difference-Based MetaprogrammingQuestions?

Expressiveness

Difference-based metaprogramming separatesanalysis from modification.

public class Example {

private int x = 0;

private int y = 0;

@@LogAndCount

public void foo() { · · · }

@@LogAndCount

public void bar() { · · · }

Injection Conflicts

M1 M2 M3

Injection Conflicts

M1 M2 M3

backstage java - making a difference in metaprogrammingzpalmer/publications/bsj... · 2016. 1....

Documents