mummer 游騰楷杜海倫 王慧芬曾俊雄 2007/01/02. outlines suffix tree mummer 1.0 mummer 2.1...
TRANSCRIPT
MUMmer
游騰楷 杜海倫王慧芬 曾俊雄
2007/01/02
Outlines
• Suffix Tree• MUMmer 1.0• MUMmer 2.1• MUMmer 3.0• Conclusion
Suffix Tree
游騰楷
Trie
• A Trie, also called a prefix tree, is a tree that records the information of strings.
• It likes a dictionary of those strings.
• Advantages comparing to BST: – Search time.
– Space.
– Prefix matching.
– Balance.
Suffix Trie and Suffix Tree
• A suffix tree, Tree(T), is a compact trie that represents all the suffixes of a string T.
Suffix Trie and Suffix Tree(cont.)
a
b
b
aaa
aa
b
b
b
a
baab
baab
ab
abaab
baab
aab
ab
b
Suffix Trie Suffix Tree
Suffix Trie and Suffix Tree(cont.)
• Let |T| = n• Suffix trie needs O(n2) space.
– Consider T = an/2bn/2
• Suffix tree needs O(n) space.– Since every symbol can cause only one branch, total no
des (edges) cannot exceed O(n).
– In a node we only record the starting position of corresponding suffix.
– In a edge we need not to record the whole substring but the starting and ending positions of the substring.
1,1
2,54,5 2,5
Suffix Trie and Suffix Tree(cont.)1:abaab
2:baab
3:aab
4:ab
5:b
Suffix Tree
1:abaab$
3:aab$
2:baab$
5:b$
2,2
3,5
a
baab
baab
ab
Suffix Tree
3,5 4:ab$
2,2
Suffix Tree: Full Text Index
• P occurs in T
P is a prefix of some suffix of T
Path for P exists in Tree(T)
Linear Time Construction
• 1973, Weiner gave the first linear time algorithm. • 1976, McCreight gave a more readable algorithm.• They were all processing from right to left.• In 1992, Ukkonen gave a left-to-right on-line algo
rithm.
On-line Construction of Suffix Trie
• We have O(n2) time to do this, it is quite easy.
• We construct Triei from Triei-1.
• We need to record the current end points of every suffix.
a
a
b
b
Trie(abaaba)
a
b
b
aa
a
b
b
aaa
aa
a
b
b
aaa
aa
b
b
b
a
b
b
aaa
aa
b
b
ba
a
a
Two Lemmas
• We call the current end points of suffix i as Ci.
1. If some Ci ever branch out it will never branch again.
2. If Ci does not branch, those Cj also does not branch for j > i.
Ideas to Find Suffix Tree
• Thus – If one branched we never consider it again.
– The branches have an order from small indices suffix to large indices.
• For a suffix we have 3 phases:1. Going along original tree.
2. Branching a new leaf.
3. Growing the leaf.
Ideas to Find Suffix Tree(cont.)
• 3 phases:1. Going along original tree.
2. Branching a new leaf.
3. Growing the leaf.
• We just record the longest suffix that is not branch yet.
• When it branches, we try to find next suffix.• To do this, we need suffix links.
Suffix Links
• Suffix link is a pointer from an internal node xS to another internal node S. x , and S = *.
• Suffix tree of “abaabab$”:
a b
aabab
ababb
ababb
ba
abaabab$baabab$
aabab$
abab$
bab$
ab$ b$
$
Using Suffix Link to Find Common Substring(1)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
Consider abaaaba
Using Suffix Link to Find Common Substring (2)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
abaaabaabaa
Using Suffix Link to Find Common Substring (3)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
abaaababaa
Using Suffix Link to Find Common Substring (4)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
abaaabaaa
Using Suffix Link to Find Common Substring (5)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
abaaaba
Using Suffix Link to Find Common Substring (6)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
abaaabaaaba
Using Suffix Link to Find Common Substring (7)
a b
a
abab
ababb
ababb
ba
abaabab$
baabab$
aabab$
abab$
bab$
ab$ b$
$
abaaaba ababaa
Construct Suffix Tree
• Like matching, we use suffix link to find next suffix.
Suffix Tree(abaabab$)
1,- 2,-
1 step
aba
1,- 2,-
ab
1,-
a
1,12,-
abaa
2,-4,-
1,12,-
abaab
2,- 4,-
1 step
1,12,-
abaaba
2,-4,-
2 step
abaabab
1,12,-
2,-4,-
Branch
abab
1,12,-
2,3 4,-
4,- 7,- Use suffix link to find this, and go down by “bab”
bab
1,12,-
2,3 4,-
4,- 7,- Nearest ancestor internal node
abab
1,12,3
2,3 4,-
4,- 7,-4,-
7,-Nearest ancestor internal node
bab
1,12,3
2,3 4,-
4,- 7,-4,-
7,-
ab
1 step
Suffix Tree(abaabab$)
1,12,2
3,34,-
4,- 7,-4,-
7,-
2,2
8,-3,38,-
abaabab$baabab$
aabab$
abab$
bab$
ab$ b$
8,- $
Time Complexity
• For phase 3, we do nothing.• For phase 1, we totally use O(n) time.• For phase 2, we do O(n) times of branch. The only
bottleneck is how fast can we find next position.
Time Complexity(cont.)
• Consider we branch at the red circle having distance t to its parent.
• Next suffix can at most pass through t internal nodes.
• Use amortized analysis, we can easily find total internal nodes the algorithm passing through is O(n).
t steps
Time Complexity(cont.)
• Thus, constructing suffix tree needs O(n) time and O(n) space.
Applications
• Longest common substring.• Repeating of a pattern in a string.• Approximate matching.• etc.
Suffix Arrayhattivatti
attivatti
ttivatti
tivatti
ivatti
vatti
atti
tti
ti
i
ε
ε
atti
attivatti
hattivatti
i
ivatti
ti
tivatti
tti
ttivatti
vatti
11
7
2
1
10
5
9
4
8
3
6Suffix array of hattivatti: (11, 7, 2, 1, 10, 5, 9, 4, 8, 3, 6)
att binary search
It can construct from suffix tree in linear time. (DFS)
Special Thanks to
• Esko Ukkonen• Hsueh-I Lu
• Wikipedia
• Some figures of this slides are based on their slides.
• Suffix Tree• MUMmer 1.0• MUMmer 2.1• MUMmer 3.0• Conclusion
MUMmer 1.0 –Alignment of whole genomes
D95922019 杜海倫2007/01/02
About The System - MUMmer
• For rapidly aligning whole genome sequences– Assumption: the sequences are closely related
– Output:• Alignment of the input sequences
• Highlighting the exact differences in the genomes
– SNPs, insertions, significant repeats, tandem repeats, reversals
– Main idea• Suffix tree
• Longest increasing subsequence (LIS)
• Smith-Waterman alignment
Alignment step
• Perform a maximal unique match (MUM) decomposition of the two genomes -> Suffix tree
• Sort the MUMs, and extract the longest possible set of matches in the same order -> LIS
• Close gaps -> Smith-Waterman alignment
• Output!
Step 1• Suffix tree
8 3
Step 1 (cont’)Construct a suffix tree T for genome A
Add the suffixes for genome B(implement: A+dummy character+B)
Find out unique matching sequence: an internal node with exactly two child nodes,
such that the child nodes are leaf nodes from different genomes
Find out MUM: For highly similar genomes, set MUM>=50bp
For more distantly related genomes, set MUM>=20bp
Step 2
• Sort, LIS=> O(KlogK) => O(N)– K: the numbers of MUMs
– K<<N/logN
Step 3• Process the gap into one of the four classes– SNP
• Genome A: cgtcataaagt
• Genome B: cgtcctaaagt
– Insert• Genome A: cgtctaaagtggggaaaactctgg
• Genome B: cgtctaaagt. . . . . . . . Ctctgg
• Transposition or simple insertions
– Polymorphic regions• Genome A: cgtctaaagtggggaaaactctgg
• Genome B: cgtctaaagta tgacaggctctgg
• Should be aligned
– Repeat• Genome A: aaggaaggaaggagct
• Genome B: aaggaagg. . . . agct
Result and Discussion
• Comparing two strain of tuberculosis– H37Rv and CDC1551– >99% identical– Be able to catalog
• all SNPs• all insertions of every length• All tandem repeat with different copy numbers
– Performance (DEC Alpha 4100)• 5s for step1• 45s for step 2• 5s for step 3
Result and Discussion (cont’)
Result and Discussion (cont’)
• Comparing two Mycoplasma genome– M.genitalium (580074nt) and M.pneumoniae (226000n
t)
– Performance (DEC Alpha 4100)• 6.5s for step1
• 0.02s for step 2
• 116s for step 3
– FASTA: many hours
Result and Discussion (cont’)
FASTA
25mers
MUMmer
Result and Discussion (cont’)
• Comparing human and mouse– 222930bp of human chromosome 12 (accession no.
U47924) and 227538bp of mouse chromosome 6 (accession no. AC002397)
– Performance -29s• 1.6s for step1
Result and Discussion (cont’)
• Suffix Tree• MUMmer 1.0• MUMmer 2.1• MUMmer 3.0• Conclusion
MUMmer 2.1
王慧芬
MUMmer 2.1• Fast algorithms for large-scale genome alignment and
comparison• by Delcher, Phillippy, Carlton and Salzberg• Nucleic Acids Research 2002• http://www.tigr.org/software/mummer/MUMmer2.pdf
Agenda
• Key in MUMmer 1• Improvements in MUMmer 2.1• Technical improvements in MUMmer 2.1• Application to DNA sequence alignment
– Alignment of incomplete genomes
MUMmer 1• Key
– Built a suffix tree containing 2 input sequences
– Find all maximal unique matches (MUMs) between them.
• MUM (Maximal Unique Matches)– A subsequence occurred in 2 exactly matching copies, once i
n each input sequence
– Cannot be extended in either direction
Improvements in MUMmer 2.1• Fast and less memory, by a factor of nearly three
• Able to align DNA or protein sequence
MUM1 MUM2
Time 74s (1GHz) 27s (1GHz)
Mem 293MB 100MB
To align 4.7 Mb genome of E. coli and
3.0Mb large chromosome of V.cholerae
Technical improvements in MUMmer 2.1
• A reduction in amount of memory used to store suffix trees– Kurtz (1999) technique is used
• An alternative algorithm to find initial exact matches• Cluster matches
Alternative to find initial exact matches
• MUMmer 1:– Built a suffix tree containing 2 input sequences
• MUMmer 2:– Chang-Lawler (1994) method is used
• running time is reduced
– Built a suffix tree storing only one sequence (reference)
– 2nd sequence (query) streamed against the suffix tree • memory usage is reduced by at least half
• once the suffix tree is built, arbitrarily long/multiple queries can be streamed.
Alternative to find initial exact matches (cont.)
• MUMmer 2 (cont.):– Identify where the query sequence would branch off from the tree, to find
all matches
– Unique match • Wherever a branch occurs at a tree position with just a single leaf beneath it
– Maximal match• Using suffix links to find next match (extended match)
• By checking the character immediately preceding the start of this match, we can determine whether it is a maximal match
• To find all maximal matches, it is in time proportional to the length of the query
Unique match
Maximal match
• suffix links is used to find extended match
Cluster matches• MUMmer 1:
– Align 2 complete sequences, no rearrangement.
– That is, to find a single longest alignment.
• MUMmer 2:– After matches are identified, the interval length between match
es are checked.
– If the interval length between matches is less than a user-defined gap length, the matches are joined into a cluster.
Alignment of incomplete genomes
• DNA sequencing• Shotgun sequencing • Terms • Finishing• NUCmer (NUCleotide MUMmer)
DNA sequencing
– Human genomes are approx. 3 billion bases.
– Sequencing machine can generate sequences for fragments in 500-600 bp long.
– In order to read DNA, genome is broken up into tiny of pieces (reads), each is read individually.
– After all pieces are read, they are assembled in the correct order.
Shotgun sequencing
• Extract DNA
• Fragment DNA
• Clone DNA
• Sequence both ends of clones– 500-600 bp each read
• Assemble– reads are assembled to reconstruct the genome
• Finish sequencing (close gaps)
Terms• Raw sequence
– Unassembled sequence reads
• Contig– Overlapping reads are joined into longer composite sequences, called contigs
• Finished sequence– Complete sequence of a genome with no gaps and an accuracy of >99.9%
• Full shotgun coverage– Genome coverage in random raw sequence required to produce finished sequ
ence, 8-10 fold (8-10X)
Finishing
• The process of– Determining the order and orientation of all the contigs, and – Generating additional sequence to fill in all the gaps between them
(closing all the gaps)
• Finishing phase is dispensed in many projects.• However, MUMmer 2.1 is used as a base to build a progra
m for the “finishing phase”– Align the multiple contigs to a completed reference genome, and – Align one set of contigs to another set of contigs
NUCmer (NUCleotide MUMmer)
• Built based on MUMmer 2.1 to develop a miltiple-contig alignment program
• 3 steps• Outputs
NUCmer – 3 steps• Step 1
– Input: 2 DNA sequences in 2 multi-fasta files representing partial or complete assemblies.
– Each DNA sequence is represented as contig sequence– Creates a map of all contig positions within each of the multi-fasta files– Concatenates the two files seperately – Runs MUMmer to find all exact matches between two genomes– These matches are mapped back to the separate contigs
NUCmer – 3 steps (cont.)
• Step 2 : Clustering– MUMs (output of step 1) are clustered together if they are separate
d in user-defined distance
• Step 3– Run a modified Smith-Waterman DP alignment to align the seque
nce between MUMs (output of step 2)
• Result– Alignment of “every sequence contig in the 1st file” to “every seq
uence contig in the 2nd file”– Order, orientation, and coverage identity percentage of contigs
• Suffix Tree• MUMmer 1.0• MUMmer 2.1• MUMmer 3.0• Conclusion
Improvements in MUMmer 3.0
d95922026 曾俊雄
Overview
• Functionality vs. Modularity
What’s New?
• Optimized suffix-tree library (rewrite)• Non-unique maximal matches (New!)• Distant matches (2.1)
Optimized Suffix-Tree
• The most significant improvement.– multi-contig query against multi-contig reference (conti
nue)
– rewrite, more compact (continue)
MUMmer 3.0, page 4
• multi-contig query against multi-contig reference– already in MUMmer 2.1 through Nucmer package
– now imported into the core
• more compact suffix-tree
• Some previous work:– Manber & Myers : 18.8n~22.4n bytes (DNA)
– K¨arkk¨ainen : 15n~18n (?)
– Crochemore and V´erin : 32.7n
– The strmat software package by Knight, Gusfield and Stoye : 24n~28n (string length at most 2^23)
• Basic Idea:– Use different data structure for different kinds of tree
nodes
– duplicated information should be removed
• Terms:– depth
– head position
abab
i
w
a
b the longest w, the smallest i
• Implementation by Stefan Kurtz :– n+5q (6n) integers !!
• for leaf node, depth and head position and some others are not required
– (3+1/16)n integers !!• more observation
a
aW
if the head position of aW is known, the head position of W will satisfy the constraint:
aW.headposition + 1 >= W.headposition
• case 1: =
• case 2: >
aW
aWaW
aW
W W
for case 1, the internal node representing aW doesn’t even need the head position information
Non-unique Maximal Matches
• 1.0 : matches must be unique– may miss
• 2.0 : uniqueness required in reference sequence– still may miss
• now, a command line to generate all maximal matches, regardless of uniqueness– in cost of very large output file
Distant Matches
• 1.0 : only 100% match is allowed– less sensitivity
• 2.1 : distant (not 100%) match is allowed– through extension packages
• 3.0 : improve
• Suffix Tree• MUMmer 1.0• MUMmer 2.1• MUMmer 3.0• Conclusion
Conclusion
• MUMmer is good at alignment between closely related species
• Distant matches are considered as MUMmer improved
• The open source License may have some advantage
• The space problem is still there