unix for poets

Download Unix for Poets

If you can't read please download the document

Post on 28-Nov-2014




3 download

Embed Size (px)



Unix for PoetsKenneth Ward Church AT&T Bell Laboratories kwc@research.att.com

Text is available like never before Dictionaries, corpora, etc. Data Collection Efforts:

ACL/DCI, BNC, CLR, ECI, EDR, ICAME, LDC Information Super Highway Roadkill:

email, bboards, faxes Billions and billions of words What can we do with it all? It is better to do something simple,

than nothing at all. You can do the simple things yourself

(DIY is more satisfying than begging for help from a computer ofcer.)


Exercises to be addressed


Count words in a text

2. Sort a list of words in various ways ascii order dictionary order rhyming order

3. Extract useful info from a dictionary 4. Compute ngram statistics 5. Make a Concordance



grep: search for a pattern (regular expression) sort uniq c (count duplicates) tr (translate characters) wc (word count) sed (edit string) awk (simple programming language) cut paste comm join


Exercise 1: Count words in a text

Input: text le (genesis) Output: list of words in the le with freq counts Algorithm

1. Tokenize (tr) 2. Sort (sort) 3. Count duplicates (uniq c)


Solution to Exercise 1

tr -sc A-Za-z \012 < genesis | sort | uniq -c

1 2 8 1 1 1 1 24 134 59

A Abel Abelmizraim Abidah Abide Abimael Abimelech Abraham Abram ...

Glue ____________________ read from input le < write to output le > pipe


Step by Step

sed 5q < genesis #Genesis 1:1 In the beginning God created the heaven and t 1:2 And the earth was without form, and void; and 1:3 And God said, Let there be light: and there w 1:4 And God saw the light, that [it was] good: an tr -sc A-Za-z \012 < genesis | sed 5q Genesis In the beginning


tr -sc A-Za-z \012 < genesis | sort | sed 5q A A Abel Abel tr -sc A-Za-z \012 < genesis | sort | uniq -c | sed 5q 1 2 8 1 1

A Abel Abelmizraim Abidah


More Counting Exercises Merge the counts for upper and lower case.

tr a-z A-Z < genesis | tr -sc A-Z \012 | sort | uniq -c

Count sequences of vowels

tr a-z A-Z < genesis | tr -sc AEIOU \012| sort | uniq -c

Count sequences of consonants

tr a-z A-Z < genesis | tr -sc BCDFGHJKLMNPQRSTVWXYZ \012 | sort | uniq -c


sort lines of text

Example Explanation _ ______________________________________sort d sort f sort n sort nr sort +1 sort +0.50 sort +1.5 dictionary order fold case numeric order reverse numeric order start with eld 1 (starting from 0) start with 50th character start with 5th character of eld 1

See man page man sort

- 10 -

Sort Exercises Sort the words in Genesis by freq

tr -sc A-Za-z \012 < genesis | sort | uniq -c | sort nr > genesis.hist Sort them by dictionary order Sort them by rhyming order (hint: rev)

1 1 5 15 1 1

... freely sorely Surely surely falsely fly ...

echo hello world | rev dlrow olleh echo hello world | rev | rev hello world

- 11 -

Important Points Thus Far

Tools: tr, sort, uniq, sed, rev Glue: , Example: count words in a text Variations tokenize by vowel, merge upper and lower case sort by freq, dictionary order, rhyming order Pipes exibility: simple yet powerful

- 12 -


Algorithm 1. tokenize by word 2. print word i and word i + 1 on the same line 3. count tr -sc A-Za-z \012 < genesis > genesis.words tail +2 genesis.words > genesis.nextwords paste genesis.words genesis.nextwords ... And God said Let ... God said Let there

- 13 -

paste genesis.words genesis.nextwords | sort | uniq -c > genesis.bigrams sort nr < genesis.bigrams | sed 5q 372 287 192 185 178 of in And And said the the he the unto

Exercise: count trigrams

- 14 -

grep & egrep: An Example of a Filter

Count ing words tr -sc A-Za-z \012 < genesis | grep ing$ | sort | uniq -c Explanation Example ________________________________________ grep gh grep con grep ing$

nd lines containing gh nd lines beginning with con nd lines ending with ing delete lines containing gh grep v gh grep v con delete lines beginning with con grep v ing$ delete lines ending with ing

- 15 -

lines with an uppercase char lines starting with an uppercase cha lines ending with an uppercase char lines with all uppercase chars grep [aeiouAEIOU] lines with a vowel grep [aeiouAEIOU] lines starting with a vowel lines ending with a vowel grep [aeiouAEIOU]$ ditto grep i [aeiou] grep i [aeiou] grep i [aeiou]$ lines starting with a non-vowel grep i [aeiou] grep i [aeiou]$ lines ending with a non-vowel lines with two or more vowels grep i [aeiou].*[aeiou] grep i [aeiou]*[aeiou][aeiou]*$ lines with exactly one vowel grep [AZ] grep [AZ] grep [AZ]$ grep [AZ]*$

Explanation Example _ __________________________________________________________

- 16 -

Regular Expressions

Explanation Example _ ______________________________________________ a [az] [AZ] [09] [0123456789] [aeiouAEIUO] [aeiouAEIOU] . $ x* x+ xy (x) match the letter a match any lowercase letter match any uppercase letter match any digit match any digit match any vowel match any letter but a vowel match any character beginning of line end of line any number of x one or more of x (egrep only) x or y (egrep only) override precedence rules (egrep only)

- 17 -

Grep Exercises


How many uppercase words are there in Genesis? Lowercase? Hint: wc -l or grep -c How many 4-letter words?


3. Are there any words with no vowels? 4. Find 1-syllable words (words with exactly one vowel) Find 2-syllable words (words with exactly two vowels)


6. Some words with two orthographic vowels have only one phonological vowel. Delete words ending with a silent e from the 2-syllable list. Delete diphthongs. 7. Find verses in Genesis with the word light. How many have two or more instances of light? Three or more? Exactly two?

- 18 -

sed (string editor)

print the rst 5 lines (quit after the 5th line)

sed 5q < genesis print up to the rst instance of a regular expression

sed /light/q genesis substitution

Explanation Example _ _________________________________ sed s/light/dark/g sed s/ly$/ly/g sed s/[ \011].*//g

simple morph prog select rst eld

- 19 -

sed exercises


Count morphs in genesis Hint: use spell -v to extract morphs, select rst eld and count echo darkness | spell -v +ness darkness


Count word initial consonant sequences: tokenize by word, delete the vowel and the rest of the word, and count Count word nal consonant sequences


- 20 -


Etymology Alfred Aho Peter Weinberger Brian Kernighan It is a general purpose programming language, though generally intended for shorter programs

(1 or 2 lines) Especially good for manipulating lines and elds

in simple ways WARNING: awk, nawk, gawk

- 21 -

Selecting Fields by Position

print the rst eld awk {print $1} cut -f1 print the second eld awk {print $2} cut -f2 print the last eld awk {print $NF} rev | cut -f1 | rev print the penultimate eld awk {print $(NF-1)} rev | cut -f2 | rev print the number of elds awk {print $NF} Exercise: sort the words in Genesis by the number of syllables (sequences of vowels)

- 22 -

Filtering by Numerical Comparison

get lines with large frequencies awk $1 > 100 {print $0} genesis.hist awk $1 > 100 {print} genesis.hist awk $1 > 100 genesis.hist Recall genesis.hist contains the words in genesis and their frequencies sed 5q genesis.hist 17052 2427 and 2407 the 1359 of 1251 And predicates: >, =, genesis.types Find palindromes rev < genesis.types | paste - genesis.types | awk $1 == $2 A I O a deed did ewe noon s A I O a deed did ewe noon s

1. == works on strings 2. paste 3.

- 24 -

Find words that can also be spelled backwards rev < genesis.types | cat - genesis.types | sort | uniq -c | awk $1 >= 2 {print $2} A I O a deed did draw evil ewe live no noon on s saw ward was Exercise: compare the bible and wsj. Find words that are in one and not in the other. Do it with awk and then do a man on comm, and do it again.

- 25 -

Filtering by Regular Expression Matching

lookup words ending in ed awk $2 /ed$/ genesis.hist grep ed$ genesis.hist count ed words (by token) awk $2 /ed$/ {x = x + $1} END {print x} genesis.hist tr -sc A-Za-z \012 < genesis | grep ed$ | wc -l count ed words (by type) awk $2 /ed$/ {x = x + 1} END {print x} genesis.hist tr -sc A-Za-z \012 < genesis | grep ed$ | sort | uniq -c | wc -l

- 26 -

count ed words both ways awk /ed$/ {token = token + $1; type = type + 1} END {print token, type} genesis.hist awk /ed$/ {token += $1; type++} END {print token, type} genesis.histAwk Exercises


Find frequent morphemes in Genesis

2. It is said that English avoids sequences of -ing words. Find bigrams where both words end in -ing. Do these count as counter-examples to the -ing -ing rule? 3. For comparisons sake, nd bigrams where both words end in -ed. Should there also be a prohibition against -ed -ed? Are there any examples of -ed -ed in Genesis? If so, how many? Which verse(s)?

- 27 -

Memory across lines

Exercise: write a uniq -c program in awk. Hint: the following almost works awk $0 == prev { c++ } $0 != prev { print c, prev c=1 prev=$0 } Fix it so it doesnt drop the last record. echo a a b 2 a 2 b 2 c