ast - the only true tool for building javascript
TRANSCRIPT
![Page 1: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/1.jpg)
AST (Abstract Syntax Tree)The only true tool for building JavaScript
![Page 2: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/2.jpg)
Source mapsEpic win in debugging
![Page 3: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/3.jpg)
Source maps – epic win in debugging
![Page 4: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/4.jpg)
Source maps – epic win in debugging
![Page 5: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/5.jpg)
BuildersEpic fail in debugging
![Page 6: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/6.jpg)
Builders – epic fail in debugging
umdify:
// UMD definition
output += '(function(root, factory) {\n';
output += ' if (typeof define === "function" && define.amd) {\n';
output += ' define([' + depNames.join(', ') + '], factory);\n';
output += ' } else if (typeof exports === "object") {\n';
output += ' module.exports = factory(require);\n';
…
![Page 7: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/7.jpg)
Builders – epic fail in debugging
grunt-amd-wrap:
var srcText = grunt.file.read(file.src[0]);
var destText = amdWrap(srcText);
![Page 8: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/8.jpg)
Builders – epic fail in debugging
gulp-concat:
buffer.push(file.contents);
…
var joinedContents = Buffer.concat(buffer);
![Page 9: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/9.jpg)
Builders – epic fail in debugging
universal-transformer:
function transform(srcText) {
return 'var answer = 42;';
}
![Page 10: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/10.jpg)
Your code is not a string
It has a soul
![Page 11: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/11.jpg)
Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7;
![Page 12: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/12.jpg)
Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7;'// Life, Universe and Everything\nvar answer = 6 * 7;'
![Page 13: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/13.jpg)
Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7;
[
{ type: "Keyword", value: "var" },
{ type: "Identifier", value: "answer" },
{ type: "Punctuator", value: "=" },
{ type: "Numeric", value: "6" },
{ type: "Punctuator", value: "*" },
{ type: "Numeric", value: "7" },
{ type: "Punctuator", value: ";" }
]
![Page 14: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/14.jpg)
Your code has a soul
[
{ type: "Keyword", value: "var" },
{ type: "Identifier", value: "answer" },
{ type: "Punctuator", value: "=" },
{ type: "Numeric", value: "6" },
{ type: "Punctuator", value: "*" },
{ type: "Numeric", value: "7" },
{ type: "Punctuator", value: ";" }
]
{
type: "Program",
body: [{
type: "VariableDeclaration",
declarations: [{
type: "VariableDeclarator",
id: {type: "Identifier", name: "answer"},
init: {
type: "BinaryExpression",
operator: "*",
left: {type: "Literal", value: 6},
right: {type: "Literal", value: 7}
}
}],
kind: "var"
}]
}
![Page 15: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/15.jpg)
Your code has a soul{
type: "Program",
body: [{
type: "VariableDeclaration",
declarations: [{
type: "VariableDeclarator",
id: {type: "Identifier", name: "answer"},
init: {
type: "BinaryExpression",
operator: "*",
left: {type: "Literal", value: 6},
right: {type: "Literal", value: 7}
}
}],
kind: "var"
}]
}
Program
VariableDeclaration
VariableDeclarator
Identifier(“answer”) BinaryExpression(*)
Literal(6) Literal(7)
![Page 16: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/16.jpg)
Your code has a soul
// Life, Universe, and Everything
var answer = 6 * 7; Program
VariableDeclaration
VariableDeclarator
Identifier(“answer”) BinaryExpression(*)
Literal(6) Literal(7)
![Page 17: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/17.jpg)
Code toolsHow can we work with code AST?
![Page 18: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/18.jpg)
Parsing
• JavaScript• SpiderMonkey: Reflect.parse – Mozilla's Parser API• Esprima – most popular ECMAScript parser in JS• Acorn – faster alternative ECMAScript parser in JS• UglifyJS – has own parser with custom AST format• Traceur – has ES6 parser that can be used separately as well• … (as a lot of language tools do) …
• CoffeeScript• CoffeeScriptRedux – rewrite of CS compiler that internally uses CS AST with conversion to JS
AST
• JSX• esprima-fb – Facebook's fork of Esprima Harmony branch• jsx-esprima – es* tools based JSX to JS AST transpiler
![Page 19: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/19.jpg)
Parsing
acorn.parse('var answer = 6 * 7;', {locations: true});
// In each node.
loc: {
start: {
line: 2,
column: 0
},
end: {
line: 2,
column: 19
}
}
![Page 20: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/20.jpg)
Linting
![Page 21: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/21.jpg)
Querying
var found;
estraverse.traverse(tree, {
enter: function (node) {
if (node.type === 'Identifier' && node.name[0] === '_') {
found = node;
return estraverse.VisitorOption.Break;
}
}
})
![Page 22: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/22.jpg)
Querying
require('grasp-equery')
.query('if ($cond) return $yes; else return $no;', ast)
![Page 23: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/23.jpg)
Querying
require('grasp-squery')
.query('if[then=return][else=return]', ast)
![Page 24: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/24.jpg)
Constructing
{ type: "Program", body: [{ type: "VariableDeclaration", declarations: [{ type: "VariableDeclarator", id: {type: "Identifier", name: "answer"}, init: { type: "BinaryExpression", operator: "*", left: {type: "Literal", value: 6}, right: {type: "Literal", value: 7} } }], kind: "var" }]}
![Page 25: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/25.jpg)
Constructing
var b = require('ast-types').builders;
b.variableDeclaration('var', [
b.variableDeclarator(
b.identifier('answer'),
b.binaryExpression(
'*',
b.literal(6),
b.literal(7)
)
)
]);
![Page 26: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/26.jpg)
Constructing
estemplate('var <%= varName %> = 6 * 7;', {
varName: {type: 'Identifier', name: 'answer'}
});
![Page 27: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/27.jpg)
Transforming
var counter = 0, map = Object.create(null);
result = estraverse.replace(tree, {
enter: function (node) {
if (node.type === 'Identifier' && node.name[0] === '_')
node.name = map[node.name] || (map[node.name] = '$' + counter++);
}
});
![Page 28: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/28.jpg)
Transforming
var Renamer = recast.Visitor.extend({
init: function () {
this.counter = 0;
this.map = Object.create(null);
},
getId: function (name) {
return this.map[name] || (this.map[name] = '$' + this.counter++);
},
visitIdentifier: function (node) {
if (node.name[0] === '_') node.name = this.getId(node.name);
}
});
![Page 29: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/29.jpg)
Generating
var output = escodegen.generate(ast, {
sourceMap: true,
sourceMapWithCode: true
});
fs.writeFileSync('out.js', output.code);
fs.writeFileSync('out.js.map', output.map.toString());
![Page 30: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/30.jpg)
Building with ASTWhat can we improve here?
![Page 31: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/31.jpg)
File-based builders (Grunt)
Parsing code
Transforming AST
Generating code
Writing file
Reading file
Plugin
![Page 32: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/32.jpg)
Transforming AST
Generating code
Parsing code
Streaming builders (Gulp, Browserify)
Reading file
Writing file
Plugin
![Page 33: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/33.jpg)
TransformingAST
Next logical step
Reading file
Writing file
Parsing code
Generating code
![Page 35: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/35.jpg)
Already available
astersrcwatchdestrunner
aster-parsejsesnextjsxcoffee
aster-changed
aster-concataster-equeryaster-squeryaster-rename-idsaster-traverseaster-uglifyaster-umd
npm keyword: aster-plugin
![Page 36: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/36.jpg)
Sample build script
aster.watch(['src/**/*.js', 'src/**/*.coffee', 'src/**/*.jsx'])
.throttle(500)
.map(changed(
src => src.map(equery({
'if ($cond) return $expr1; else return $expr2;':
'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
}))
))
.map(concat('built.js'))
.map(umd({exports: 'superLib'}))
.map(aster.dest('dist', {sourceMap: true}))
.subscribe(aster.runner);
![Page 37: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/37.jpg)
Plugins – reactive AST transformers
module.exports = source => {
source = source || 'built.js';
return files => files
.flatMap(file => Rx.Observable.fromArray(file.program.body))
.toArray()
.map(body => ({
type: 'File',
program: {type: 'Program', body},
loc: {source}
}));
};
![Page 38: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/38.jpg)
Integration with generic build systems
grunt.initConfig({
aster: {
options: {
equery: {
'if ($cond) return $expr1; else return $expr2;':
'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
},
concat: 'built.js',
umd: {exports: 'superLib'},
dest: {sourceMap: true}
},
files: { 'dist': ['src/**/*.js', 'src/**/*.coffee', 'src/**/*.jsx'] }
}
});
![Page 39: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/39.jpg)
Integration with generic build systems
gulp.src(['src/**/*.js', 'src/**/*.coffee', 'src/**/*.jsx'])
.pipe(aster(src => src
.map(equery({
'if ($cond) return $expr1; else return $expr2;':
'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
})
.map(concat('built.js'))
.map(umd({exports: 'superLib'}))
))
.pipe(gulp.dest('dist'))
![Page 40: AST - the only true tool for building JavaScript](https://reader036.vdocuments.site/reader036/viewer/2022062312/555a441bd8b42a83368b5259/html5/thumbnails/40.jpg)
Questions?https://github.com/asterjs