gilad bracha ministry of truth embedding dsls in newspeak: newshell, ebnf and hopscotch
TRANSCRIPT
Gilad BrachaMinistry of Truth
Embedding DSLs in Newspeak:
NewShell, EBNF and Hopscotch
Three DSLs
Shell Scripting
Parsing
UI
Newspeak Syntax for Java
Programmers
Newspeak Syntax for Java
Programmers
Newspeak Syntax for C#
Programmers
Syntax: Unary Expressions
x foo
means
x.foo()
Syntax: Unary Expressions
x foo
means
x.foo()
Binary Expressions
6 + 3 factorial
evaluates to 12.
In javanese, we’d write
6.+(3.factorial()) or, really
6.plus(3.factorial())
Keyword Expressions
Windows version: 7 + 0i .
is equivalent to all these:
Windows version: (7 + 0i).
Windows version: (7 + (0 i)).
Windows.version(7 + 0.i()) ;
Keyword Expressions
add: 1 to: Cobol
in javanese would be
add(1, Cobol);
Big advantage - no arity errors
Three DSLs
Shell Scripting
Parsing
UI
NewShell
class NewShell = (
ls value: ‘-l’.
ls - ‘lt’.
echo value: ‘Hello World’.
svn value: ‘ls’ value: ‘svn://myDir’.
(ls value findTokens: {Character cr}) collect: ls.
)()
Three DSLs
Shell Scripting
Parsing
UI
BNF
id = letter (letter | digit) *
Parser Combinators
BNF
id = letter (letter | digit) *
Newspeak
id = letter, (letter | digit) star.
Parser Combinators
BNF
id = letter (letter | digit) *
Newspeak
id = letter, (letter | digit) star.
Javanese
id = letter().seq(letter().or(digit()).star());
Parser Combinators
How it Works
id = letter, (letter | digit) star.
How it Works
id = letter, (letter | digit) star.
How it Works
id = letter, (letter | digit) star.
How it Works
id = letter, (letter | digit) star.
How it Works
id = letter, (letter | digit) star.
letter
How it Works
id = letter, (letter | digit) star.
letter
How it Works
id = letter, (letter | digit) star.
letter
How it Works
id = letter, (letter | digit) star.
letter
How it Works
id = letter, (letter | digit) star.
letter
How it Works
id = letter, (letter | digit) star.
letter
How it Works
id = letter, (letter | digit) star.
letter
letter
How it Works
id = letter, (letter | digit) star.
letter
letter
How it Works
id = letter, (letter | digit) star.
letter
digitletter
How it Works
id = letter, (letter | digit) star.
letter
|
digitletter
How it Works
id = letter, (letter | digit) star.
letter star
|
digitletter
How it Works
id = letter, (letter | digit) star.
letter star
|
digitletter
,
Why is this Pretty?
id = letter, (letter | digit) star.
Why is this Ugly?
id = letter().seq(letter().or(digit()).star());
Why is this Ugly?
id = letter().seq(letter().or(digit()).star());
vs.
id = letter (letter | digit) *
vs.
id = letter, (letter | digit) star.
Why is this Ugly?
id = letter().seq(letter().or(digit()).star());
vs.
id = letter (letter | digit) *
vs.
id = letter, (letter | digit) star.
Why is it Ugly?
A programming language is low level when its programs require attention to the irrelevant
- Alan Perlis
class ExampleGrammar1 = ExecutableGrammar ( |
digit = charBetween: ‘0’ and: ‘9’.
letter = (charBetween: ‘a’ and:’z’) | (charBetween: ‘A’ and:’Z’).
id = letter, (letter | digit) star.
identifier = tokenFor: id.
hat = tokenFromChar: ‘ˆ’.
expression = identifier.
returnStatement = hat, expression.
| )()
A Complete Grammar
returnStatement = hat, expression wrapper:[:r : e | ReturnStatAST on: e ]
Building an AST
Building an AST
returnStatement = hat, expression
wrapper:[:r : e |
ReturnStatAST on: e
]wrapper: ,
expression
^
returnStatement = hat, expression
wrapper:[:r : e |
ReturnStatAST on: e
]
Grammar
returnStatement = hat, expression
wrapper:[:r : e |
ReturnStatAST on: e
]
Semantic Action
Superclass (pure grammar specification):
returnStatement = hat, expression
Subclass (AST builder):
returnStatement = (
super returnStatement
wrapper:[:r : e | ReturnStatAST on: e] )
Factor out Grammar
class ExampleParser1= ExampleGrammar1 () (
id = (
ˆsuper id
wrapper:[:fst :snd | fst asString, (String withAll: snd)]
)
identifier = (
ˆsuper identifier
wrapper:[:v | VariableAST new name: v token;
start: v start; end: v end].
)
r
Modular Parser
…
returnStatement = (
ˆsuper returnStatement
wrapper:[:r :e | ReturnStatAST new expr:e;
start: r start; end: e end].
)
)
Modular Parser
class ExampleGrammar2 = ExampleGrammar1 (
|
if = tokenFromSymbol:#if.
then = tokenFromSymbol:#then.
else = tokenFromSymbol:#else.
ifStatement = if, expression, then, statement, else, statement.
statement = ifStatement | returnStatement.
|
)()
Extending a Grammar
Mutual RecursionIfStmtreturnStat
|
if then else
stmt
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit nilnilnilnilnilnilnil
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsernilnilnilnilnilnil
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parser
nilnilnilnilnil
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parser
nilnilnilnil
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parser
nilnilnil
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parsera parser
nilnil
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parsera parsera parsera parser
nil
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parser
nilnilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parser
a parsernilnil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parser
a parsera parser
nil
nil
Mutual Recursion
statement
ifStatement
letterid
identifierhat
expressionreturnStatement
if
thenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parser
a parsera parserboom!
nil
Addressed with laziness in Haskell
Can be addressed with closures
ifStatement = if, [expression], [then], [statement],
[else], [statement].
statement = ifStatement | [returnStatement].
Mutual Recursion
Addressed with laziness in Haskell
Can be addressed with closures
ifStatement = if, [expression], [then], [statement],
[else], [statement].
statement = ifStatement | [returnStatement].
Mutual Recursion
Use Reflection instead of Laziness.
Framework initializes all slots to “stand-in” parsers which act as forward references
Being Reflective rather than Lazy
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit fwd reffwd reffwd reffwd reffwd reffwd reffwd ref
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parserfwd reffwd reffwd reffwd reffwd reffwd ref
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parserfwd reffwd reffwd reffwd reffwd ref
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parserfwd reffwd reffwd reffwd ref
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parserfwd reffwd reffwd ref
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parserfwd reffwd ref
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parsera parsera parser
fwd reffwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parserfwd reffwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parsera parserfwd reffwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parsera parsera parserfwd reffwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parsera parsera parsera parserfwd ref
Mutual Recursion
statementifStatement
letterid
identifierhat
expressionreturnStatement
ifthenelse
digit a parsera parsera parsera parsera parsera parsera parser
a parsera parsera parsera parsera parser
Mutual RecursionIfStmtreturnStat
|
if then else
stmt
FwdRef
Being Reflective rather than Lazy
Use Reflection instead of Laziness.
Framework initializes all slots to “stand-in” parsers which act as forward references
When grammar is completely initialized, stand-ins are connected to originals and forward calls to them
Mutual RecursionIfStmtreturnStat
|
if then else
stmt
FwdRef
The Mirror is Mightier Than the Sloth
Grammar1
Grammar2 Parser1
Parser2
Grammar1
Grammar2 Parser1
Parser2
Grammar1
Grammar2
Parser1
Parser2
class ExampleParser2 = ExampleParser1 mixin | >
ExampleGrammar2 ()
(
ifStatement = (
ˆsuper ifStatement
wrapper:[:ifKw :e :thenKw :s1 :elseKw :s2 | …
]
)
)
Composing Parsers
Shareable among several IDEs/compilers that require distinct ASTs
Shareable among other language tools: syntax colorizers, postprocessing tools
Grammar is shared executable
specification
Three DSLs
Shell Scripting
Parsing
UI
Hopscotch
A GUI application framework
Higher level of abstraction than widgets
Provides DSL for composing presentations out of pieces known as fragments
Fragments
Fragments of presentation
Atomic widgets correspond to leaf fragments
Aggregates represented via compound fragments
Composed via combinators
Fragment Combinators
blank, filler
button:action:, link:action:, image:, menuWithLabelsAndActions:
row:, column:, centered:, collapsed:expanded:
deferred:, draggable:subject:
Dispatching up the fragment hierarchy
Dispatching up the visual containment hierarchy makes sense
Old idea (at least since NewtonScript)
Realized using doesNotUnderstand:
Demo
Connections
Self & Smalltalk
ConnectionsParser Combinators.
Too much to survey; some highlights:
Parsec (Haskell parser combinator library)
Swierstra et al.
Sparsec (Scala parser combinator library)
PEGs (Ford, POPL04, ICFP)
Connections
Parser Combinators (continued)
Tedir (Plesner-Hansen’s masters thesis)
Ometa (OOPLA 07 DLS)
Conclusions
Newspeak supports DSLs via
Unconventional syntax
keywords
operators as methods
implicit receivers
Dynamic typing and Reflection
doesNotUnderstand:
Conclusions
Newspeak also supports DSLs via
Mixins, Modules, Hierarchy inheritance
IDE
Combinators are a powerful technique
Compositional
Useful in libraries and languages
Status Available at http://newpeaklanguage.org
Open source under Apache 2.0 license
Work in Progress
Expect some tweaks to syntax and semantics
Implementation still not complete - especially libraries
Credits
Peter Ahe
Vassili Bykov
Yaron Kashai
Bill Maddox
Eliot Miranda
This file is licensed under the Creative Commons Attribution ShareAlike 3.0 License. In short: you are free to share and make derivative works of the file under the conditions that you appropriately attribute it, and that you distribute it only under a license identical to this one. Official
license.
The Newspeak eye used in the bullets, slide background etc. was designed by Victoria Bracha and is used by permission.