![Page 1: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/1.jpg)
How to Create a High-Speed Template Engine in Python
makoto kuwatahttp://www.kuwata-lab.com/
PyconMini JP 2011
Pythonにおけるテンプレートエンジンの高速化と失敗談
![Page 2: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/2.jpg)
Profile お前、誰よ?@makotokuwata
http://www.kwuata-lab.com/
Ruby/PHP/Python programmer
Creator of Erubis (*)
Python4PHPer
パイを広げるのに熱心なだけの人間(*) default template engine on Rails 3
![Page 3: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/3.jpg)
Python Products
Tenjin : very fast temlate engine
Kook : task utility like Ant/Rake
Benchmarker : a good friend for performance
Oktest : new-style testing library
![Page 4: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/4.jpg)
Tenjin
Very fast
One file, 2000 lines
Full-featured
Python 3 support
Google App Engine
Release 1.0 coming soon!
http://www.kuwta-lab.com/tenjin/
<table> <?py for x in xs: ?> <tr> <tr>${x}</tr> </tr> <?py #endfor ?></table>
![Page 5: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/5.jpg)
BenchmarkTenjin
Mako
Jinja2
Templetor
Cheetah
Django
Genshi
Kid0 600 1200 1800 2400 3000
34.6
55.7
114.2
562.3
903.0
1257.6
1426.4
2660.1
pages/secPython 2.5.5, MacOS X 10.6 (x86_64), 2GBTenjin 1.0.0, Mako 0.2.5, Jinja2 2.2.1, Templetor 0.32,Cheetah 2.2.2, Django 1.1.0, Genshi 0.5.1, Kid 0.9.6
![Page 6: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/6.jpg)
Benchmarks forString Concatenation
![Page 7: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/7.jpg)
append()
_buf = []_buf.append(s)_buf.append(s)_buf.append(s)output = "".join(_buf)
![Page 8: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/8.jpg)
Benchmarkappend()
0 200 400 600 800 1000
pages/sec
![Page 9: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/9.jpg)
extend()
_buf = []_buf.extend((s, s, s, ))output = "".join(_buf)
![Page 10: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/10.jpg)
Benchmarkappend()
extend()
0 200 400 600 800 1000
pages/sec
![Page 11: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/11.jpg)
StringIO
from cStringIO import StringIO_buf = StringIO()_buf.write(s)_buf.write(s)_buf.write(s)output = _buf.getvalue()
![Page 12: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/12.jpg)
Benchmarkappend()
extend()
StringIO
0 200 400 600 800 1000
pages/sec
![Page 13: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/13.jpg)
mmap
import mmap_buf = mmap.mmap(-1, 2*1024*1024)_buf.write(s)_buf.write(s)_buf.write(s)length = _buf.tell()_buf.seek(0)output = _buf.read(length)
![Page 14: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/14.jpg)
Benchmarkappend()
extend()
StringIO
mmap
0 200 400 600 800 1000
pages/sec
![Page 15: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/15.jpg)
Generator
def _gen(s): yield s yield s yield s
output = "".join(_gen(s))
![Page 16: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/16.jpg)
Benchmarkappend()
extend()
StringIO
mmap
generator
0 200 400 600 800 1000
pages/sec
![Page 17: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/17.jpg)
Slice
_buf = [""]_buf[-1:] = (s, s, s, "")output = "".join(_buf)# or_buf = []_buf[999999:] = (s, s, s, )output = "".join(_buf)
![Page 18: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/18.jpg)
Benchmarkappend()
extend()
StringIO
mmap
generator
slice[-1:]
slice[99999:]
0 200 400 600 800 1000
pages/sec
![Page 19: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/19.jpg)
Bound method
_buf = []_extend = _buf.extend_extend((s, s, s, ))output = "".join(_buf)
![Page 20: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/20.jpg)
Benchmarkappend()
extend()
StringIO
mmap
generator
slice[-1:]
slice[99999:]
extend() (bound)0 200 400 600 800 1000
pages/sec
![Page 21: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/21.jpg)
Summary
Fast
bound method >= slice[] > extend()
Slow
Generator > append() > mmap > StringIO
![Page 22: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/22.jpg)
Try Benchmark Script
$ wget wget http://pypi.python.org/packages/source/B/Benchmarker/Benchmarker-3.0.0.tar.gz$ tar xzf Benchmarker-3.0.0.tar.gz$ cd Benchmarker-3.0.0/$ sudo python setup.py install$ cd examples/$ python bench_strconcat.py
![Page 23: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/23.jpg)
Step by Step toTune-up Template Code
![Page 24: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/24.jpg)
HTML Template<html> <head> : <table> <?py for item in items: ?> <tr> <td>#{item}</td> : </tr> <?py #endfor ?> </table> :
![Page 25: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/25.jpg)
Python Code_buf = []; _buf.append("<html>\n") :for item in items: _buf.append(" <tr>\n") _buf.append(" <td>"); _buf.append(item); _buf.append("<td>\n"); :#endfor :return "".join(_buf)
![Page 26: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/26.jpg)
Benchmarkappend (singleline)
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 27: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/27.jpg)
Multiple Line String## before_buf.append("<!DOCTYPE HTML>\n")_buf.append("<html>\n")_buf.append(" <head>\n")
## after_buf.append("""<!DOCTYPE HTML><html> <head>""")
Eliminates method call
![Page 28: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/28.jpg)
Benchmarkappend (singleline)append (multiline)
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 29: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/29.jpg)
From append() to extend()## before_buf.append(""" <td>""")_buf.append(item)_buf.append("""</td>\n""")
## after_buf.extend((""" <td>""", item, """</td>\n""", ))
Eliminates method call
![Page 30: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/30.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 31: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/31.jpg)
Bound Method## before_buf.extend(("...", "...", "...", ))_buf.extend(("...", "...", "...", ))_buf.extend(("...", "...", "...", ))
## after_extend = _buf.extend_extend(("...", "...", "...", ))_extend(("...", "...", "...", ))_extend(("...", "...", "...", ))
Eliminates fetch method
![Page 32: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/32.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 33: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/33.jpg)
str() function## before_extend((" <td>", item1, """</td> <td>""", item2, """</td> <td>""", item3, """</td>""", ))
## after_extend((" <td>", str(item1), """</td> <td>""", str(item2), """</td> <td>""", str(item3), """</td>""", ))
Necessary in Python!
![Page 34: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/34.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
extend + str
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 35: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/35.jpg)
Local Variable## before_extend((str(item1), str(item2), str(item3), ))
## after_str = str_extend((_str(item1), _str(item2), _str(item3), ))
Local var is faster than global/build-in var
![Page 36: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/36.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
extend + strextend + _str=str
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 37: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/37.jpg)
Format ('%' operator)## before_extend(("<td>", _str(item1), """</td><td>""", _str(item2), """</td><td>""", _str(item3), "</td>\n", ))
## after_append("""<td>%s</td><td>%s</td><td>%s</td>\n""" % \ (item1, item2, item3, ))
Delete all str() callby '%' operator
![Page 38: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/38.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
extend + strextend + _str=strappend + format
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 39: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/39.jpg)
None => Empty String## afterdef to_str(v): if v is None: return "" else: return str(v)
_to_str = to_str_extend((_to_str(item1), _to_str(item2), _to_str(item3), ))
Converts None to empty string
![Page 40: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/40.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
extend + strextend + _str=strappend + format
extend + to_strextend + _to_str=to_str
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 41: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/41.jpg)
Escape HTML## afterdef escape_html(s): return s.replace('&', '&') \ .replace('<', '<') \ .replace('>', '>') \ .replace('"', '"')
_extend((""" <td>""", escape_html(to_str(item)), """</td>\n""", ))
![Page 42: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/42.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
extend + strextend + _str=strappend + format
extend + to_strextend + _to_str=to_str
escape_html + strescape_html + to_str
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 43: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/43.jpg)
C Extension## afterfrom webext import to_str, escape_html
_extend((""" <td>""", escape_html(to_str(item)), """</td>\n""", ))## or_extend((""" <td>""", escape_html(item), """</td>\n""", ))
Implemented in C
webext: http://pypi.python.org/pypi/Webext/
![Page 44: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/44.jpg)
Benchmarkappend (singleline)append (multiline)extend (unbound)
extend (bound)
extend + strextend + _str=strappend + format
extend + to_strextend + _to_str=to_str
escape_html + strescape_html + to_str
webext.escape_html, to_strwebext.escape_html
0 2000 4000 6000 8000 10000 12000
pages/sec
![Page 45: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/45.jpg)
Extreme join()
## after_buf = [ "<td>", item1, "</td>\n<td>", item2, "</td>\n<td>", "", item3, "", "</td>\n",]output = webext.join(_buf, escape=webext.escape_html)
Not escaped if index % 2 == 0
Be escaped if index % 2 == 1
(no need to callescape_html() !)
![Page 46: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/46.jpg)
Benchmark
Not implemeted yet...
![Page 47: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/47.jpg)
Summary
String concatenation is not a bottleneck
extend() & join() are enough fast
Bottleneck is str() and escape_html()
join() should call str() internally
C Extension (webext) is great
![Page 48: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/48.jpg)
Other Topics
![Page 49: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/49.jpg)
Google says...
... The major web applications we have surveyed have indicated that they bottleneck primarily on template systems, ...
http://code.google.com/p/unladen-swallow/wiki/ProjectPlan
Django?
![Page 50: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/50.jpg)
Case Study #1http://www.myweightracker.com/
Switch from Django template to Tenjin
DjangoM, C, Network, etc...
M, C, Network, etc...App Speed
30% Up!
https://groups.google.com/group/kuwata-lab-products/browse_thread/thread/b50877a9c56d64c9/60f77b5c9b9f5238
![Page 51: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/51.jpg)
Case Study #2Ruby on Rails 1.2
Remove helper methods by preprocessing
Helper MethodsM, C, Network, etc...
M, C, Network, etc...App Speed
100% Up!
http://jp.rubyist.net/magazine/?0021-Erubis
template engine
![Page 52: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/52.jpg)
Components of View Layer
Cache Mechanism
Template Engine
Helper Functions
Important for performance! More Important
for performance!
Just one of them
![Page 53: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/53.jpg)
Preprocessing in Tenjin
<p>${_('Hello')}</p>
_extend(("<p>", _('Hello'), "</p>", ))
<p>こんにちは</p>
Convert
Execute Called everytime
![Page 54: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/54.jpg)
Preprocessing in Tenjin
<p>${{_('Hello')}}</p>
_extend(("<p>こんにちは</p>", ))
<p>こんにちは</p>
Convert
Execute
Call functionin this stage
Func call removed
![Page 55: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/55.jpg)
Python v.s. Others
plTenjin (Perl)
pyTenjin+Webext
phpTenjin (PHP)
pyTenjin (Python)
rbTenjin (Ruby)
0 2500 5000 7500 10000 12500
2634.8
2682.9
2788.0
4179.7
12108.0
pages/sec
Perl is the
Champion!
![Page 56: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/56.jpg)
Why Perl is so Fast?
No need to call str(val) nor val.toString()
Bytecode op for string concatenation
## slowvar $x = join("", ($s1, $s2, $3, ... ));
## extremely fast!var $x = $s1 . $s2 . $s3 . ...;
![Page 57: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/57.jpg)
C Ext v.s. Pure Script
plTenjinMobaSiF
Template::Toolkit
pyTenjin+WebextpyTenjinCheetah
rbTenjineruby
0 2500 5000 7500 10000 12500
pages/sec
C ExtC Ext
C Ext
C Ext
Pure Perl
Pure Python
Pure Ruby
Python + C Ext
No need to impl engine in C
(except helpers)
![Page 58: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/58.jpg)
Summary
View layer components
Template engine, Helper functions, and Cache mechanism
No need to implement engine in C (except helper functions)
Perl is great
Django temlate engine sucks
![Page 59: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/59.jpg)
Appendix
Tenjin: fast & full-featured template engine
http://www.kuwata-lab.com/tenjin/
Webext: C extension for escape_html()
http://pypi.python.org/pypi/Webext/
Benchmarker: a utility for benchmarking
http://pypi.python.org/pypi/Benchmarker/
![Page 60: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/60.jpg)
Appendix
Cより速いRubyプログラムhttp://www.kuwata-lab.com/presen/rubykaigi2007.pdf
http://jp.rubyist.net/magazine/?0022-FasterThanC
Javaより速いLL用テンプレートエンジンhttp://www.kuwata-lab.com/presen/LL2007LT.pdf
テンプレートシステム入門http://jp.rubyist.net/magazine/?0024-TemplateSystemhttp://jp.rubyist.net/magazine/?0024-TemplateSystem2
![Page 61: How to Create a High-Speed Template Engine in Python](https://reader036.vdocuments.site/reader036/viewer/2022081505/554a2b38b4c90520578b4d43/html5/thumbnails/61.jpg)
thank you