writing dsls with parslet - wicked good ruby conf

Writing DSLswith Parslet

Jason Garber

W i c k e d G o o d R u b y C o n f

TDDTATFT

TDDTATFT

Agile! ScrumPair

Programming

VIM

PomodoroContinuousIntegration

Continuous Delivery

Parsing

DSLSDomain-Specific Languages

<?xml version="1.0"?> <configuration><configSections><sectionGr

oup name="userSettings" type="System.Configuration.UserSettings

Group, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken

=b77a5c561934e089"><section name="MSDNSampleSettings.My.MySetti

ngs" type="System.Configuration.ClientSettingsSection, System,

Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e0

89" allowExeDefinition="MachineToLocalUser" requirePermission="

false"/></sectionGroup></configSections> ... <userSettings><MSD

NSampleSettings.My.MySettings><setting name="Setting" serialize

As="String"><value>SomeDefaultValue</value></setting></MSDNSamp

leSettings.My.MySettings></userSettings> </configuration>

</xml>

I DSLs

RssReader::Application.routes.draw do devise_for :users

resources :feeds, only: [:edit, :update, :show, :index] post '/feeds', to: 'feeds#create', as: 'create_feed' resources :users do get 'page/:page', action: :index, on: :collection end resources :posts do member do put :update_category end end get '/my_profile', to: 'users#my_profile', as: :my_profile root to: "home#home"end

describe Stack do context "stack with one item" do let(:stack) { a_stack_with_one_item }

context "when popped" do before { stack.pop } it { should be_empty } end endend

click_on "Sign Up"fill_in "Email", with: account[:email]fill_in "Password", with: account[:password]fill_in "Confirmation", with: account[:password_confirmation]fill_in "Name", with: account[:name]select account[:birthyear].to_s, from: "Year born"check "Terms"click_on "I'm ready to join!"current_path.should eq "/accounts/#{account.id}/dashboard"page.should have_content "Dashboard"

desc 'Generate markup and stylesheets and open browser preview'task :preview => FileList['*.html'] + FileList['*.css'] do |t| sh 'open -g index.html'end rule '.html' => '.haml' do |t| puts "Rebuilding #{t.name}" sh "haml #{t.source} #{t.name}"end rule '.css' => lambda { |cssfile| source(cssfile) } do |t| puts "Rebuilding #{t.name}" sh "sass #{t.source} #{t.name}"end

get '/' do @posts = Post.all(:order => [:id.desc], :limit => 20) erb :indexend

get '/post/new' do erb :newend

get '/post/:id' do @post = Post.get(params[:id]) erb :postend

post '/post/create' do post = Post.new(params) status 201 redirect "/post/#{post.id}"end

rule(:table) do (str("table") >> attributes?.as(:attributes) >> str(".\n")).maybe >> table_row.repeat(1).as(:content) >> block_end end rule(:table_row) do table_row_attributes >> table_row_content >> end_table_row end rule(:table_row_attributes) { (attributes?.as(:attributes) >> str(". ")).maybe } rule(:table_row_content) { (table_header | table_data).repeat(1).as(:content) } rule(:end_table_row) { str("|") >> (block_end.present? | (str("\n"))) } rule(:table_header) do str("|_. ") >> table_content.as(:content) end rule(:table_data) do str("|") >> str("\n").absent? >> td_attributes? >> table_content.as(:content) end

Internal DSLsFluent Interfaces

EXTERNALDSLS

upstream puma { server unix:///tmp/sockets/puma.sock fail_timeout=0;} server { listen 80 default deferred; server_name promptworks.com www.promptworks.com; root /srv/promptworks/public; charset utf-8; if (-f $document_root/system/maintenance.html) { return 503; } error_page 503 @maintenance; location @maintenance { rewrite ^(.*)$ /system/maintenance.html last; break; }}

http://www.promptworks.com

http://www.promptworks.com

<([A-Z][A-Z0-9]*)\b[^>]*>(.*?)<\/\1>

\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3} (?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b

SELECT DISTINCT sc1.id FROM ( SELECT DATE_ADD(entry_point.start_date, INTERVAL (digits_1.digit * 10 + digits_2. (1 << (DAYOFWEEK(DATE_ADD(entry_point.start_date, INTERVAL (digits_1.digit * 10 + FROM (SELECT CAST('2012-03-01' AS date) AS start_date) AS entry_point INNER JOIN (SELECT 0 AS digit UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION ON (digits_1.digit * 10 + digits_2.digit) <= (DATEDIFF(DATE_ADD(entry_point.start ) AS md INNER JOIN schedules AS sc1 INNER JOIN negative_link_influences AS neg on (sc1.call_type_id = neg.call_type_id or neg.call_type_id = -1) WHERE sc1.schedule_on = md.date AND neg.affected_shift = 0;

BEGIN{ TOTAL=0 }{ TOTAL = TOTAL + $1 }END{ print TOTAL/NR }

{print "<li><a href=\"" $1 "\">" $1 "</a></li>"}

! doctype htmlhtml head title Test body - unless items.empty? ol - items.each do |item| li = item - else p No items

Feature: Searching music As a User I want to be able to search music So I can play it Background: Given I am logged in Scenario: Search music Given I am on the search screen When I search for "mix" Then I should see the mixtape

class erlang { file { "/etc/apt/sources.list.d/esl-erlang.list": ensure => present, owner => root, content => 'deb http://example.com/debian precise contrib'; } exec { "apt-update": command => "/usr/bin/apt-get update", refreshonly => true; } package { "esl-erlang": ensure => installed, require => Exec['apt-update', 'import-key']; }}include erlang

http://binaries.erlang-solutions.com/debian

http://binaries.erlang-solutions.com/debian

Heroku buildpack: Ruby ====================== This is a [Heroku buildpack](http://devcenter.heroku.com/articles/buildpacks) forIt uses [Bundler](http://gembundler.com) for dependency management. Usage ----- ### Ruby Example Usage: $ heroku create --stack cedar --buildpack https://github.com/heroku/heroku-bu $ git push heroku master

http://devcenter.heroku.com/articles/buildpacks

http://devcenter.heroku.com/articles/buildpacks

http://gembundler.com

http://gembundler.com

https://github.com/heroku/heroku-bu

https://github.com/heroku/heroku-bu

title = "T??? Example"

[owner]name = "Tom Preston-Werner"organization = "GitHub"bio = "GitHub Cofounder & CEO\nLikes tater tots and beer."dob = 1979-05-27T07:32:00Z # First class dates? Why not?

[database]server = "192.168.1.1"ports = [ 8001, 8001, 8002 ]connection_max = 5000enabled = true

global= { \time 4/4 \key c \major \tempo "Allegro" 4 = 132 \set Score.skipBars = ##t}

Violinone = \new Voice { \relative c''{ \set Staff.midiInstrument = #"violin" \set tupletSpannerDuration = #(ly:make-moment 1 4)

% Jason solo R1 * 9 r2 r4 \times 2/3 { c4( b8)

gsave1 0.5 scale70 100 48 0 360 arcfillgrestore/Helvetica-Bold 14 selectfont1.0 setgray29 45 moveto(Hello, world!) showshowpage

.accordion { li { border-top: 1px solid #e2e4e6; &:first-child { border-top-color: transparent; } } a { @include rem(padding, 5px 10px 6px); &.icon { padding-left: 40px; position: relative; } &.icon img { @include rem(left, 10px); margin-top: -2px; position: absolute; } }}

h1. Give Textile a try!

A *simple* paragraph with a line break,some _emphasis_ and a "link":http://redcloth.org

* an item* and another

# one# two# three

http://redcloth.org

http://redcloth.org

A_HLGN = /(?:\<(?!>)|\<\>|\=|[()]+)/A_VLGN = /[\-^~]/C_CLAS = '(?:$[^)]+$)'C_LNGE = '(?:\[[^\]]+\])'C_STYL = '(?:\{[^}]+\})'S_CSPN = '(?:\\\\\d+)'S_RSPN = '(?:/\d+)'A = "(?:#{A_HLGN}?#{A_VLGN}?|#{A_VLGN}?#{A_HLGN}?)"S = "(?:#{S_CSPN}?#{S_RSPN}|#{S_RSPN}?#{S_CSPN}?)"C = "(?:#{C_CLAS}?#{C_STYL}?#{C_LNGE}?|#{C_STYL}?#{C_LNGE}?#{C_CLAS}?|#{C_LNGE}?#{C_STYL}?#{C_CLAS}?)"

PUNCT = Regexp::quote( '!"#$%&\'*+,-./:;=?@\\^_`|~' )HYPERLINK = '(\S+?)([^\w\s/;=\?]*?)(\s|$)'

def links( text ) text.gsub!( / ([\s\[{(]|[#{PUNCT}])? # $pre " # start (#{C}) # $atts ([^"]+?) # $text \s? (?:$([^)]+?)$(?="))? # $title ": (\S+?) # $url (\/)? # $slash ([^\w\/;]*?) # $post (?=\s|\.[#{PUNCT}]+|$) /x ) do |m| pre,atts,text,title,url,slash,post = $~[1..7]

url = check_refs( url )

atts = pba( atts ) atts << " title=\"#{ title }\"" if title atts = shelve( atts ) if atts

"#{ pre }<a href=\"#{ url }#{ slash }\"#{ atts }>" + "#{ text }</a>#{ post }" endend

“Some people, when confronted with a problem, think ‘I know, I'll use regular expressions.’Now they have two problems.”

— Jamie Zawinski

%%{ machine superredcloth_scan; include superredcloth_common "ext/superredcloth_scan/superredcloth_common.rl"; action extend { extend = rb_hash_aref(regs, ID2SYM(rb_intern("type"))); } # blocks notextile_tag_start = "<notextile>" ; notextile_tag_end = "</notextile>" CRLF? ; notextile_block_start = ( "notextile" >A %{ STORE(type) } A C :> "." ( "." %extend | "" ) " "+ ) ; pre_tag_start = "<pre" [^>]* ">" (space* "<code>")? ; pre_tag_end = ("</code>" space*)? "</pre>" CRLF? ; pre_block_start = ( "pre" >A %{ STORE(type) } A C :> "." ( "." %extend | "" ) " "+ ) ; bc_start = ( "bc" >A %{ STORE(type) } A C :> "." ( "." %extend | "" ) " "+ ) ; btype = ( "p" | "h1"... block_start = ( btype >A %{ STORE(type) } A C :> "." ( "." %extend | "" ) " "+ ) ; next_block_start = ( btype A C :> "."+ " " ) >A @{ p = reg - 1; } ; pre_tag := |* pre_tag_end { CAT(block); DONE(block); fgoto main; }; default => esc_pre; *|; notextile_tag := |* notextile_tag_end { DONE(block); fgoto main; }; default => cat; *|; }%%

void rb_str_cat_escaped(VALUE str, char *ts, char *te, unsigned int opts);void rb_str_cat_escaped_for_preformatted(VALUE str, char *ts, char *te, unsigned int opts);VALUE superredcloth_inline(VALUE, char *, char *, VALUE);VALUE superredcloth_inline2(VALUE, VALUE, VALUE);

#define CAT(H) rb_str_cat(H, ts, te-ts)#define CLEAR(H) H = rb_str_new2("")#define INLINE(H, T) rb_str_append(H, rb_funcall(rb_formatter, rb_intern(#T), 1, regs))

VALUE superredcloth_transform(rb_formatter, p, pe, refs) VALUE rb_formatter; char *p, *pe; VALUE refs;{ if (RSTRING(block)->len > 0) { ADD_BLOCK(); }

if ( NIL_P(refs) && rb_funcall(refs_found, rb_intern("empty?"), 0) == Qfalse ) { return superredcloth_transform(rb_formatter, orig_p, orig_pe, refs_found); } else { rb_funcall(rb_formatter, rb_intern("after_transform"), 1, html); return html; }}


/* * redcloth_scan.c.rl * * Copyright (C) 2009 Jason Garber */#define redcloth_scan_c

#define RSTRING_NOT_MODIFIED#include <ruby.h>#include "redcloth.h"

VALUE mRedCloth, super_ParseError, super_RedCloth, super_HTML, super_LATEX;VALUE SYM_escape_preformatted, SYM_escape_attributes;

#line 23 "ext/redcloth_scan/redcloth_scan.c"static const unsigned char _redcloth_scan_actions[] = { 0, 1, 0, 1, 2, 1, 3, 1, 4, 1, 5, 1, 6, 1, 7, 1, 9, 1, 10, 1, 11, 1, 12, 1, 13, 1, 14, 1, 15, 1, 16, 1, 17, 1, 18, 1, 19, 1, 20, 1, 21, 1, 22, 1, 23, 1, 24, 1, 25, 1, 26, 1, 27, 1, 28, 1, 29, 1, 30, 1, 34, 1, 35, 1, 36, 1, 38, 1, 40, 1, 42, 1, 43, 1, 44, 1, 45, 1, 48, 1, 57, 1, 58, 1, 59, 1, 60, 1, 61, 1, 62, 1, 63, 1, 64, 1, 65, 1, 66, 1, 70, 1, 73, 1, 74, 1, 75, 1, 76, 1, 77, 1,

grammar Arithmetic rule additive multitive ( '+' multitive )* end

rule multitive primary ( [*/%] primary )* end

rule primary '(' additive ')' / number end

rule number '-'? [1-9] [0-9]* endend

require 'parslet'

class MiniParser < Parslet::Parser rule(:integer) { match('[0-9]').repeat(1) } root(:integer)end

MiniParser.new.parse("1324321")# => "1324321"@0

Parslet::Atoms

str('Boston')

Parslet::Atoms

str('Boston').parse('Boston')

Parslet::Atoms

str('Boston').parse('Boston')# => "Boston"@0

Parslet::Atoms


match('[0-9a-f]')

Parslet::Atoms


match('[0-9a-f]')

any

Operators

str('Wicked') >> str('Good')

str('Ruby') | str('Elixir')

match('[Bb]') >> str('oston') | match('[Mm]') >> str('assachusetts')

Repetition

str('foo').repeatstr('foo').repeat(1)str('foo').repeat(1,3)str('foo').repeat(0, nil)str('foo').maybe

Presence

str('Java') >> str('Script').present?

str('0').repeat(1).absent? >> match('[\d]').repeat(1)

1. Create a grammarWhat should be legal syntax?

2. Annotate the grammar:What is important data?

3. Create a transformation:How do I want to work with that data?

Capture

str('Common').parse('Common')# => "Common"@0

str('Common').as(:park).parse('Common')# => {:park=>"Common"@0}

Capture

str('a').repeat.as(:b)# => {:b=>"aaa"@0} str('a').as(:b).repeat# => [{:b=>"a"@0}, {:b=>"a"@1}, {:b=>"a"@2}] str('a').as(:a) >> str('b').as(:b) >> str('c')# => {:a=>"a"@0, :b=>"b"@1}

table(#prices).| Adults | $5 || Children | $2 |

<table id="prices"> <tr> <td>Adults</td> <td>$5</td> </tr> <tr> <td>Children</td> <td>$2</td> </tr></table>

def parenthesized(atom) str('(') >> atom >> str(')')end parenthesized(match['\d']).parse("(500)")

class HtmlTag < Parslet::Parser root(:tag) rule(:tag) { open_tag | close_tag | self_closing_tag | comment_tag } rule(:open_tag) { str("<") >> tag_name >> attributes? >> str(">") } rule(:close_tag) { str("</") >> tag_name >> str(">") } rule(:tag_name) { match("[A-Za-z_:]") >> name_char.repeat } ...end

class BlockHtmlTag < HtmlTag rule(:tag_name) do inline_tag_name.absent? >> any_tag_name end

rule(:inline_tag_name) do INLINE_TAGS.map {|name| str(name) }.reduce(:|) endend

> HtmlTag.new.open_tag.methods => [:name, :block, :try, :parslet, :to_s_inner, :parse, :apply, :cached?, ...]

> HtmlTag.new.open_tag.parse("<blockquote>") => "<blockquote>"@0

> HtmlTag.new.open_tag.parse("</blockquote>")

Parslet::ParseFailed: Failed to match sequence ('<' TAG_NAME ATTRIBUTES? '>') at line 1 char 2. from /Users/jasongarber/.rvm/gems/ruby-2.0.0-p247/gems/parslet-1.5.0/lib/parslet/cause.rb:63:in `raise' from /Users/jasongarber/.rvm/gems/ruby-2.0.0-p247/gems/parslet-1.5.0/lib/parslet/atoms/base.rb:46:in `parse' from (irb):8 from /Users/jasongarber/.rvm/rubies/ruby-2.0.0-p247/bin/irb:16:in `<main>'

begin RedClothParslet::Parser::BlockHtmlTag.new.tag.parse("<img>")rescue Parslet::ParseFailed => failure puts failure.cause.ascii_treeend

Expected one of [OPEN_TAG, CLOSE_TAG, SELF_CLOSING_TAG, COMMENT_TAG] at line 1 char 1.|- Failed to match sequence ('<' TAG_NAME ATTRIBUTES? '>') at line 1 char 2.| `- Failed to match sequence ((!INLINE_TAG_NAME / &(INLINE_TAG_NAME NAME_CHAR{1, })) ANY_TAG_NAME) at line 1 char 2.| `- Expected one of [!INLINE_TAG_NAME, &(INLINE_TAG_NAME NAME_CHAR{1, })] at line 1 char 2.| |- Input should not start with INLINE_TAG_NAME at line 1 char 2.| `- Input should start with INLINE_TAG_NAME NAME_CHAR{1, } at line 1 char 2.|- Failed to match sequence ('</' TAG_NAME '>') at line 1 char 1.| `- Expected "</", but got "<i" at line 1 char 1.|- Failed to match sequence ('<' TAG_NAME ATTRIBUTES? SPACES? '/' '>') at line 1 char 2.| `- Failed to match sequence ((!INLINE_TAG_NAME / &(INLINE_TAG_NAME NAME_CHAR{1, })) ANY_TAG_NAME) at line 1 char 2.| `- Expected one of [!INLINE_TAG_NAME, &(INLINE_TAG_NAME NAME_CHAR{1, })] at line 1 char 2.| |- Input should not start with INLINE_TAG_NAME at line 1 char 2.| `- Input should start with INLINE_TAG_NAME NAME_CHAR{1, } at line 1 char 2.`- Failed to match sequence ('<!--' (!COMMENT_TAG_END .){0, } COMMENT_TAG_END) at line 1 char 1. `- Expected "<!--", but got "<img" at line 1 char 1.

I Objects

describe RedClothParslet::Parser::HtmlTag do it { should parse("<div>") } it { should parse("<hr />") } it { should parse("</div>") } it { should parse("") }

describe "attribute" do subject { described_class.new.attribute } it { should parse(" class='awesome'") } it { should_not parse(' 9kittens="cute"') } endend

$ rspec spec/parser/html_tag_spec.rb

1) RedClothParslet::Parser::HtmlTag tag Failure/Error: it { should parse("</div>") } expected TAG to be able to parse "</div>" Expected one of [OPEN_TAG, CLOSE_TAG, SELF_CLOSING_TAG, COMMENT_TAG] at line 1 char 1. |- Failed to match sequence ('<' TAG_NAME ATTRIBUTES? '>') at line 1 char 2. | `- Failed to match sequence ([A-Za-z_:] NAME_CHAR{0, }) at line 1 char 2. | `- Failed to match [A-Za-z_:] at line 1 char 2. |- Failed to match sequence ('<' TAG_NAME '>') at line 1 char 2. | `- Failed to match sequence ([A-Za-z_:] NAME_CHAR{0, }) at line 1 char 2. | `- Failed to match [A-Za-z_:] at line 1 char 2. |- Failed to match sequence ('<' TAG_NAME ATTRIBUTES? SPACES? '/' '>') at line 1 char 2. | `- Failed to match sequence ([A-Za-z_:] NAME_CHAR{0, }) at line 1 char 2. | `- Failed to match [A-Za-z_:] at line 1 char 2. `- Failed to match sequence ('<!--' (!COMMENT_TAG_END .){0, } COMMENT_TAG_END) at line 1 char 1. `- Expected "<!--", but got "</di" at line 1 char 1. # ./spec/parser/html_tag_spec.rb:9:in `block (3 levels) in <top (required)>'

13/13: 100% |==========================================| Time: 00:00:00

Finished in 0.01879 seconds13 examples, 1 failure

1. Create a grammarWhat should be legal syntax?

2. Annotate the grammar:What is important data?

3. Create a transformation:How do I want to work with that data?

Transformationstree = {left: {int: '1'}, op: '+', right: {int: '2'}} class T < Parslet::Transform rule(int: simple(:x)) { Integer(x) }end

T.new.apply(tree) # => {:left=>1, :op=>"+", :right=>2}

Transformationstree = {left: {int: '1'},

op: '+', right: {int: '2'}}

class T < Parslet::Transform rule(int: simple(:x)) { Integer(x) } rule(op: '+', left: simple(:l), right: simple(:r)) { l + r }endT.new.apply(tree) # => 3

Transformations

rule(:content => subtree(:c), :attributes => subtree(:a)) do |dict| {:content => dict[:c], :opts => RedCloth::Ast::Attributes.new(dict[:a])}end

rule(:table => subtree(:a)) do RedCloth::Ast::Table.new(a[:content], a[:opts])end

rule(:bq => subtree(:a)) do RedCloth::Ast::Blockquote.new(a[:content], a[:opts])end

Who you callin’ slow?

class ErbParser < Parslet::Parser rule(:ruby) { (str('%>').absent? >> any).repeat.as(:ruby) }

rule(:expression) { (str('=') >> ruby).as(:expression) } rule(:comment) { (str('#') >> ruby).as(:comment) } rule(:code) { ruby.as(:code) } rule(:erb) { expression | comment | code }

rule(:erb_with_tags) { str('<%') >> erb >> str('%>') } rule(:text) { (str('<%').absent? >> any).repeat(1) }

rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text) } root(:text_with_ruby)end








rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text root(:text_with_ruby)end








rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text root(:text_with_ruby)end

include Parslet::Acceleratoroptimized = apply( parser, rule( (str(:x).absent? >> any).repeat ) { GobbleUp.new(x, 0) })

Custom Atoms

• Parse a limited set of natural-language queries

• “Who makes the best cheesesteaks in Boston?”

• EngTagger: a corpus-trained, probabilistic tagger

• Custom Parslet::Atom

rule(:question) do interrogative.maybe.as(:int) >> verb_phrase.maybe.as(:verb) >> superlative_phrase.maybe.as(:sup) >> subject.as(:subj) >> prepositional_phrase.repeat.as(:preps) >> sentence_end.repeat.as(:punct)end

rule(:verb_phrase) { (verb_present | verb_past | verb_future).as(:vp) }rule(:verb_present) { word(:VBZ, :VB).as(:present) }rule(:verb_past) { word(:VBD).as(:past) }rule(:verb_future) { word(:MD).as(:future) >> word(:VB).maybe.as(:infinitive) }

Other Crazy Uses

Other Crazy Uses

•User-supplied formulas / logic

Other Crazy Uses


•Logs

Other Crazy Uses


•Logs

•Streaming text

Other Crazy Uses


•Logs

•Streaming text

•The Right Reverend and Right Honourable the Lord Bishop of London Richard John Carew Chartres

Who will write the next awesome DSL?

Who will write the next awesome DSL!.

You

Conclusions

Conclusions• DSLs make life better


• internal_dsl > external_dsl if internal_dsl.practicable?



• Keep your parser clean



• Keep your parser clean

• “Situational awareness!”

@jasongarber @promptworks

[email protected]

mailto:[email protected]

mailto:[email protected]

writing dsls with parslet - wicked good ruby conf

Technology

end end end

end end rule

category end end

content end rule

row end rule

index end

new end

home end