jug tutorial: coarse-level parallel programming in python
TRANSCRIPT
![Page 1: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/1.jpg)
Jug Tutorial: Coarse-Level Parallel Programming inPython
Luis Pedro Coelho
May 15, 2013
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (1 / 24)
![Page 2: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/2.jpg)
Problem
Brute force a password.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (2 / 24)
![Page 3: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/3.jpg)
Assumptions
The crypt module exists with elements:decrypt: decrtypt ciphertext given a password.isgood: test whether this text could be the plaintext.letters: just the letters.
Also, we know that the password is five letters.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (3 / 24)
![Page 4: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/4.jpg)
Simple Solution
import itertoolsfrom crypt import decode, letters, isgood, preprocess
ciphertext = file('secret.msg').read()ciphertext = preprocess(ciphertext)
for p in itertools.product(letters, repeat=5):text = decode(ciphertext, p)if isgood(text):
passwd = "".join(map(chr,p))print '%s:%s' % (passwd, text)
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (4 / 24)
![Page 5: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/5.jpg)
This solution does not take advantage of multiple processors.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (5 / 24)
![Page 6: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/6.jpg)
Tasks
import itertoolsfrom crypt import decode, letters, isgood, preprocess
ciphertext = file('secret.msg').read()ciphertext = preprocess(ciphertext)
def decrypt(ciphertext, p):text = decode(ciphertext, p)if isgood(text):
passwd = "".join(map(chr,p))return (passwd, text)
# else: return None
results = []for p in itertools.product(letters, repeat=5):
results.append(Task(decrypt,ciphertext,p))
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (6 / 24)
![Page 7: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/7.jpg)
Python Magic
from jug import TaskGeneratorimport itertoolsfrom crypt import decode, letters, isgood, preprocess
ciphertext = file('secret.msg').read()ciphertext = preprocess(ciphertext)
@TaskGeneratordef decrtypt(ciphertext, p):
text = decode(ciphertext, p)if isgood(text):
passwd = "".join(map(chr,p))return (passwd, text)
# else: return None
results = []for p in itertools.product(letters, repeat=5):
results.append(decrypt(ciphertext,p))
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (7 / 24)
![Page 8: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/8.jpg)
Enter Jug
You give it the Jugfile, it runs the tasks for you!
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (8 / 24)
![Page 9: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/9.jpg)
Jug Loop
while len(tasks) > 0:ready = [t for t in tasks if can_run(t)]for t in ready:
if not is_running(t):t.run()
tasks.remove(t)
Except jug is much fancier!
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (9 / 24)
![Page 10: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/10.jpg)
Jug Advantages
1 Automatic task-level parallelization with dependency tracking.2 Remember all intermediate results.3 Makes writing parallel code look like writing sequential code.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (10 / 24)
![Page 11: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/11.jpg)
This example is actually not so good.We have 265 ≈ 11M tasks, all of which run very fast.As a rule of thumb, your tasks should take at least a couple of seconds.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (11 / 24)
![Page 12: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/12.jpg)
Solution each task will be:Given a letter, try all passwords beginning with that letter.Now, we have 26 tasks. Much better.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (12 / 24)
![Page 13: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/13.jpg)
@TaskGeneratordef decrypt(prefix):
res = []for suffix in product(letters, repeat=5-len(prefix)):
passwd = np.concatenate([prefix, suffix])text = decode(ciphertext, passwd)if isgood(text):
passwd = "".join(map(chr, passwd))res.append( (passwd, text) )
return res
@TaskGeneratordef join(partials):
return list(chain(*partials))
results = join([decrypt([p]) for p in letters])
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (13 / 24)
![Page 14: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/14.jpg)
Let’s call jug now.(Assuming our code is in a file called jugfile.py)
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (14 / 24)
![Page 15: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/15.jpg)
$jug statusTask name Waiting Ready Finished Running-----------------------------------------------jugfile.join 1 0 0 0jugfile.decrypt 0 26 0 0...............................................Total: 1 26 0 0
Some tasks are ready. None are running.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (15 / 24)
![Page 16: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/16.jpg)
$jug execute &[1] 29501$jug execute &[2] 29502
Executing in the background. . .
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (16 / 24)
![Page 17: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/17.jpg)
$jug statusTask name Waiting Ready Finished Running-----------------------------------------------jugfile.join 1 0 0 0jugfile.decrypt 0 24 0 2...............................................Total: 1 24 0 2
Two tasks running. Good.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (17 / 24)
![Page 18: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/18.jpg)
Wait a few minutes. . .
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (18 / 24)
![Page 19: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/19.jpg)
$jug statusTask name Waiting Ready Finished Running-----------------------------------------------jugfile.join 1 0 0 0jugfile.decrypt 0 14 10 2...............................................Total: 1 14 10 2
Ten tasks have finished.Notice how the join task must wait for all the others.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (19 / 24)
![Page 20: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/20.jpg)
A few more minutes. . .
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (20 / 24)
![Page 21: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/21.jpg)
$jug statusTask name Waiting Ready Finished Running-----------------------------------------------jugfile.join 0 0 1 0jugfile.decrypt 0 0 26 0...............................................Total: 0 0 27 0
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (21 / 24)
![Page 22: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/22.jpg)
What Now?
All tasks are finished!How do I get to the results?
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (22 / 24)
![Page 23: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/23.jpg)
import jugjug.init('jugfile', 'jugdata')import jugfileresults = jug.task.value(jugfile.results)for p,t in results:
print "%s\n\n Password was '%s'" % (t,p)
jug.init is necessary to initialise the jug backend.Then, the jugfile can be directly imported as a Python module.
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (23 / 24)
![Page 24: Jug Tutorial: Coarse-Level Parallel Programming in Python](https://reader030.vdocuments.site/reader030/viewer/2022021006/62038747da24ad121e4a7720/html5/thumbnails/24.jpg)
For more information
http://luispedro.org/software/jughttp://github.com/luispedro/jug
Luis Pedro Coelho ([email protected]) ? Jug Tutorial 0 ? May 15, 2013 (24 / 24)