ch09 combinatorialpatternmatching

137
Combinatorial Pattern Matching

Upload: bioinformaticsinstitute

Post on 26-Jun-2015

111 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Ch09 combinatorialpatternmatching

Combinatorial

Pattern Matching

Page 2: Ch09 combinatorialpatternmatching

Section 1:

Repeat Finding and

Hash Tables

Page 3: Ch09 combinatorialpatternmatching

Genomic Repeats

• Repeat: A sequence of DNA that occurs more than once in a

genome.

• Example: ATGGTCTAGGTCCTAGTGGTC

Page 4: Ch09 combinatorialpatternmatching

Genomic Repeats

• Repeat: A sequence of DNA that occurs more than once in a

genome.

• Example: ATGGTCTAGGTCCTAGTGGTC

Page 5: Ch09 combinatorialpatternmatching

Genomic Repeats

• Repeat: A sequence of DNA that occurs more than once in a

genome.

• Example: ATGGTCTAGGTCCTAGTGGTC

Page 6: Ch09 combinatorialpatternmatching

Genomic Repeats

• Repeat: A sequence of DNA that occurs more than once in a

genome.

• Example: ATGGTCTAGGTCCTAGTGGTC

Page 7: Ch09 combinatorialpatternmatching

Genomic Repeats

• Repeat: A sequence of DNA that occurs more than once in a

genome.

• Example: ATGGTCTAGGTCCTAGTGGTC

• Why do we want to find repeats?

1. Understand more about evolution.

2. Many tumors are characterized by an explosion of repeats.

3. Genomic rearrangements are often associated with repeats.

Page 8: Ch09 combinatorialpatternmatching

Genomic Repeats

• Note: Often, a repeat will refer to a segment of DNA that

occurs very often with slight modifications.

• As we might imagine, this is a more difficult computational

problem.

Page 9: Ch09 combinatorialpatternmatching

Genomic Repeats

• Note: Often, a repeat will refer to a segment of DNA that

occurs very often with slight modifications.

• As we might imagine, this is a more difficult computational

problem.

• Example: Say our target segment is GGTC and we allow one

substitution:

• ATGGTCTAGGACCTAGTGTTC

Page 10: Ch09 combinatorialpatternmatching

Genomic Repeats

• Note: Often, a repeat will refer to a segment of DNA that

occurs very often with slight modifications.

• As we might imagine, this is a more difficult computational

problem.

• Example: Say our target segment is GGTC and we allow one

substitution:

• ATGGTCTAGGACCTAGTGTTC

Page 11: Ch09 combinatorialpatternmatching

Genomic Repeats

• Note: Often, a repeat will refer to a segment of DNA that

occurs very often with slight modifications.

• As we might imagine, this is a more difficult computational

problem.

• Example: Say our target segment is GGTC and we allow one

substitution:

• ATGGTCTAGGACCTAGTGTTC

Page 12: Ch09 combinatorialpatternmatching

Genomic Repeats

• Note: Often, a repeat will refer to a segment of DNA that

occurs very often with slight modifications.

• As we might imagine, this is a more difficult computational

problem.

• Example: Say our target segment is GGTC and we allow one

substitution:

• ATGGTCTAGGACCTAGTGTTC

Page 13: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Long repeats are difficult to find, but short repeats are easy.

• Simple approach to finding long repeats:

1. Find exact repeats of short l-mers (l is usually 10 to 13).

2. Use l-mer repeats to potentially extend into longer,

maximal repeats.

Page 14: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

Page 15: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

Page 16: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 17: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 18: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 19: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 20: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 21: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 22: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

Page 23: Ch09 combinatorialpatternmatching

Extending l-mer Repeats

• Example: We start with a repeat of length 4.

• GCTTACAGATTCAGTCTTACAGATGGT

• Extend both these 4-mers:

• GCTTACAGATTCAGTCTTACAGATGGT

• Maximal repeat: CTTACAGAT

Page 24: Ch09 combinatorialpatternmatching

Maximal Repeats: Issue

• In order to find maximal repeats in this way, we need ALL

start locations of ALL l-mers in the genome.

• Hashing lets us find repeats quickly in this manner.

Page 25: Ch09 combinatorialpatternmatching

Hashing

• Hashing is a very efficient way to store and retrieve data.

• What does hashing do?

• Say we are given a collection of data.

• For different entry, generate a unique integer and store the

entry in an array at this integer location.

Page 26: Ch09 combinatorialpatternmatching

Hashing: Definitions

• Records: data stored in a hash table.

• Keys: Integers identifying set of records.

• Hash Table: The array of keys used in hashing.

• Hash Function: Assigns each record to a key.

• Collision: Occurs when more than one record is mapped to the

same index in the hash table.

Page 27: Ch09 combinatorialpatternmatching

Hashing: Example

• Records: Animals

• Keys: Where each

animal eats.

Page 28: Ch09 combinatorialpatternmatching

Hashing DNA sequences

• Each l-mer can be translated into a binary string:

• A can be represented as 00

• T can be represented as 01

• G can be represented as 10

• C can be represented as 11

• Example: ATGCTA = 000110110100

• After assigning a unique integer to each l-mer, it is easy to

obtain all start locations of each l-mer in a genome.

Page 29: Ch09 combinatorialpatternmatching

Hashing: Maximal Repeats

• To find repeats in a genome:

1. For all l-mers in the genome, note the start position and the

sequence.

2. Generate a hash table index for each unique l-mer

sequence.

3. At each index of the hash table, store all genome start

locations of the l-mer that generated the index.

4. Anywhere there are collisions in the table, extend

corresponding l-mers to maximal repeats.

• How do we deal with collisions?

Page 30: Ch09 combinatorialpatternmatching

Hashing: Collisions

• In order to deal with

collisions:

• ―Chain‖ all start locations of

l-mers.

• This can be done via a data

structure called a linked list.

Page 31: Ch09 combinatorialpatternmatching

Section 2:

Exact Pattern Matching

Page 32: Ch09 combinatorialpatternmatching

Pattern Matching

• What if, instead of finding repeats in a genome, we want to

find all sequences in a database that contain a given pattern?

• This leads us to a different problem, the Pattern Matching

Problem.

Page 33: Ch09 combinatorialpatternmatching

Pattern Matching Problem

• Goal: Find all occurrences of a pattern in a text.

• Input:

• Pattern p = p1…pn of length n

• Text t = t1…tm of length m

• Output: All positions 1< i < (m – n + 1) such that the n-letter

substring of text t starting at i matches the pattern p.

• Motivation: Searching database for a known pattern.

Page 34: Ch09 combinatorialpatternmatching

Exact Pattern Matching: A Brute Force Algorithm

PatternMatching(p,t)

1 n length of pattern p

2 m length of text t

3 for i 1 to (m – n + 1)

4 if ti…ti+n-1 = p

5 output i

Page 35: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

Exact Pattern Matching: An Example

Page 36: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 37: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 38: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 39: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 40: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 41: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 42: Ch09 combinatorialpatternmatching

• PatternMatching algorithm for:

• Pattern GCAT

• Text AGCCGCATCT

• AGCCGCATCT

• GCAT

Exact Pattern Matching: An Example

Page 43: Ch09 combinatorialpatternmatching

• PatternMatching runtime: O(nm)

• In the worst case, we have to check n characters at each of

the m letters of the text.

• Example: Text = AAAAAAAAAAAAAAAA, Pattern = AAAC

• This is rare; on average, the run time is more like O(m).

• Rarely will there be close to n comparisons at each step.

• Better solution: suffix trees…solve in O(m) time.

• First we need keyword trees.

Exact Pattern Matching: Running Time

Page 44: Ch09 combinatorialpatternmatching

Section 3:

Keyword Trees

Page 45: Ch09 combinatorialpatternmatching

Keyword Trees: Example

• Keyword tree:

• Apple

Page 46: Ch09 combinatorialpatternmatching

Keyword Trees: Example

• Keyword tree:

• Apple

• Apropos

Page 47: Ch09 combinatorialpatternmatching

Keyword Trees: Example

• Keyword tree:

• Apple

• Apropos

• Banana

Page 48: Ch09 combinatorialpatternmatching

Keyword Trees: Example

• Keyword tree:

• Apple

• Apropos

• Banana

• Bandana

Page 49: Ch09 combinatorialpatternmatching

Keyword Trees: Example

• Keyword tree:

• Apple

• Apropos

• Banana

• Bandana

• Orange

Page 50: Ch09 combinatorialpatternmatching

Keyword Trees: Properties

• Stores a set of keywords in a

rooted labeled tree.

• Each edge is labeled with a

letter from an alphabet.

• Any two edges coming out of

the same vertex have distinct

labels.

• Every keyword stored can be

spelled on a path from root to

some leaf.

• Furthermore, every path from

root to leaf gives a keyword.

Page 51: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―appeal‖

• appeal

Page 52: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―appeal‖

• appeal

Page 53: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―appeal‖

• appeal

Page 54: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―appeal‖

• appeal

Page 55: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―apple‖

• apple

Page 56: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―apple‖

• apple

Page 57: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―apple‖

• apple

Page 58: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―apple‖

• apple

Page 59: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• Thread ―apple‖

• apple

Page 60: Ch09 combinatorialpatternmatching

Multiple Pattern Matching (MPM) Problem

• Goal: Given a set of patterns and a text, find all occurrences of

any of patterns in text.

• Input: k patterns p1,…,pk, and text of m characters t = t1…tm

• Output: Positions 1 < i < m where the substring of text t

starting at i matches one of the patterns pj for 1 < j < k.

• Motivation: Searching database for known multiple patterns.

Page 61: Ch09 combinatorialpatternmatching

MPM: Naïve Approach

• We could always solve MPM as k ―Pattern Matching

Problems.‖

• Runtime: Using the PatternMatching algorithm k times gives

O(kmn):

• m = length of the text.

• n = average pattern length.

Page 62: Ch09 combinatorialpatternmatching

MPM: Keyword Tree Approach

• Alternatively we could use a keyword approach.

• Let N = total length of all patterns.

• We can build the keyword tree in O(N) time.

• Total Runtime:

• With naive threading: O(N + nm)

• Aho-Corasick algorithm: Improvement to O(N + m)

Page 63: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• To match patterns in a text using a

keyword tree:

• Build keyword tree of patterns

• ―Thread‖ the text through the

keyword tree.

Page 64: Ch09 combinatorialpatternmatching

Keyword Trees: Threading

• To match patterns in a text using a

keyword tree:

• Build keyword tree of patterns

• ―Thread‖ the text through the

keyword tree.

• Threading is ―complete‖ when we

reach a leaf in the keyword tree.

• When threading is ―complete,‖

we‘ve found a pattern in the text.

Page 65: Ch09 combinatorialpatternmatching

Section 4:

Suffix Trees

Page 66: Ch09 combinatorialpatternmatching

Suffix Trees = Collapsed Keyword Trees

• Suffix Tree: Similar to

keyword tree, except edges

that form paths are

collapsed.

• Each edge is labeled with

a substring of the text.

• All internal edges have at

least two outgoing edges.

• Leaves are labeled by the

index of the pattern.

Keyword Tree Suffix Tree

Page 67: Ch09 combinatorialpatternmatching

Suffix Trees = Collapsed Keyword Trees

ATCATGTCATGCATGATGTGG

Keyword Tree Suffix Tree

Page 68: Ch09 combinatorialpatternmatching

Suffix Tree of a Text

Keyword

Tree

Suffix

Tree

• The suffix trees of a text is constructed for all its suffixes.

• Example: Suffix tree of ATCATG:

ATCATGTCATGCATGATGTGG

Page 69: Ch09 combinatorialpatternmatching

Suffix Trees: Example

• Example: Suffix tree of TCATACATGGCATACATGG

Page 70: Ch09 combinatorialpatternmatching

Suffix Trees: Runtime

ATCATGTCATGCATGATGTGG

Keyword

Tree

Suffix

Tree

• Say the length of the text is m.

• Construction of the keyword tree takes O(m2) time.

• Constructing the suffix tree takes O(m) time.

• So our method takes O(m2 + m) = O(m2) time.

• However, there exists a method of calculating the suffix tree in

O(m) time without needing the keyword tree for the suffixes.

Page 71: Ch09 combinatorialpatternmatching

Suffix Trees: Runtime

ATCATGTCATGCATGATGTGG

Keyword

Tree

Suffix

Tree

• Say the length of the text is m.

• Construction of the keyword tree takes O(m2) time.

• Constructing the suffix tree takes O(m) time.

• So our method takes O(m2 + m) = O(m2) time.

• However, there exists a method of calculating the suffix tree in

O(m) time without needing the keyword tree for the suffixes.

Page 72: Ch09 combinatorialpatternmatching

Pattern Matching with Suffix Trees

• To find any pattern in a text:

1. Build suffix tree for text of length m—O(m) time

2. Thread the pattern of length n through the suffix tree of the

text—O(n) time.

• Therefore the runtime of the Pattern Matching Problem is only

O(m + n)!

Page 73: Ch09 combinatorialpatternmatching

Pattern Matching with Suffix Trees

SuffixTreePatternMatching(p,t)

1. Build suffix tree for text t

2. Thread pattern p through suffix tree

3. if threading is complete

4. output positions of all p-matching leaves in the tree

5. else

6. output “Pattern does not appear in text”

Page 74: Ch09 combinatorialpatternmatching

Threading ATG through a Suffix Tree

Page 75: Ch09 combinatorialpatternmatching

Multiple Pattern Matching: Summary

• Keyword and suffix trees are used to find patterns in a text.

• Keyword trees:

• Build the keyword tree of patterns, and thread the text

through it.

• Suffix trees:

• Build the suffix tree of the text, and thread the patterns

through it.

Page 76: Ch09 combinatorialpatternmatching

Section 5:

Heuristic Similarity Search

Algorithms

Page 77: Ch09 combinatorialpatternmatching

Approximate vs. Exact Pattern Matching

• So far all we‘ve seen exact pattern matching algorithms.

• Usually, because of mutations, it makes sense to find

approximate pattern matches.

• Biologists often use fast heuristic approaches (rather than local

alignment) to find approximate matches.

Page 78: Ch09 combinatorialpatternmatching

Heuristic Similarity Searches

• Genomes are huge: Smith-Waterman quadratic alignment

algorithms are too slow.

• Alignment of two sequences usually has short identical or

highly similar fragments.

• Many heuristic methods (e.g. BLAST) are based on the same

idea of filtration:

• Find short exact matches, and use them as seeds for

potential match extension.

• ―Filter‖ out positions with no extendable matches.

Page 79: Ch09 combinatorialpatternmatching

Dot Matrices

• Dot matrices show

similarities between two

sequences.

• FASTA makes an implicit

dot matrix from short exact

matches, and tries to find

long approximate diagonals

(allowing for some

mismatches).

Page 80: Ch09 combinatorialpatternmatching

Dot Matrices: Example

• Identify diagonals above a

threshold length.

• Diagonals in the dot matrix

indicate exact substring

matching.

Page 81: Ch09 combinatorialpatternmatching

Diagonals in Dot Matrices

• Extend diagonals and try to

link them together, allowing

for minimal

mismatches/indels.

• Linking exact diagonals

reveals approximate

matches over longer

substrings.

Page 82: Ch09 combinatorialpatternmatching

Section 6:

Approximate String

Matching

Page 83: Ch09 combinatorialpatternmatching

Approximate Pattern Matching Problem

• Goal: Find all approximate occurrences of a pattern in a text.

• Input: A pattern p = p1…pn, text t = t1…tm, and k, the

maximum allowable number of mismatches.

• Output: All positions 1 < i < (m – n + 1) such that ti…ti+n-1 and

p1…pn have at most k mismatches. i.e.,

HammingDistance(ti…ti+n-1, p) ≤ k.

Page 84: Ch09 combinatorialpatternmatching

Approximate Pattern Matching: Brute Force

ApproximatePatternMatching(p, t, k)

1. n length of pattern p

2. m length of text t

3. for i 1 to m – n + 1

4. dist 0

5. for j 1 to n6. if ti+j-1 ≠ pj

7. dist dist + 1

8. if dist < k

9. output i

Page 85: Ch09 combinatorialpatternmatching

Approximate Pattern Matching: Running Time

• The brute force algorithm runs in O(mn) time.

• Landau-Vishkin algorithm: Gives improvement to O(km).

• We will next generalize the ―Approximate Pattern Matching

Problem‖ into a ―Query Matching Problem.‖

• Rather than patterns, we want to match substrings in a

query to substrings in a text with at most k mismatches.

• Motivation: We want to see similarities to some gene, but we

may not know which parts of the gene to look for.

Page 86: Ch09 combinatorialpatternmatching

Section 7:

Query Matching and

Filtration

Page 87: Ch09 combinatorialpatternmatching

Query Matching Problem

• Goal: Find all substrings of a query that approximately match

the text.

• Input: Query q = q1…qw

Text t = t1…tm,

n = length of matching substrings

k = maximum number of mismatches

• Output: All pairs of positions (i, j) such that the n-letter

substring of query q starting at i approximately matches the n-

letter substring of text t starting at j, with at most k mismatches.

Page 88: Ch09 combinatorialpatternmatching

Approximate Pattern Matching vs Query Matching

Page 89: Ch09 combinatorialpatternmatching

Filtration in Query Matching: Main Idea

• Approximately matching strings share some perfectly

matching substrings.

• Instead of searching for approximately matching strings

(difficult) search for perfectly matching substrings (easy).

Page 90: Ch09 combinatorialpatternmatching

Filtration in Query Matching

• We want all n-matches between a query and a text with up to k

mismatches.

• ―Filter‖ out positions we know do not match between text and

query.

• Filtration involves two processes:

1. Potential match detection: Find all exact matches of l-

tuples in query and text for some small l.

2. Potential match verification: Verify each potential match

by extending it to the left and right, until (k + 1)

mismatches are found or we have n – k matches.

Page 91: Ch09 combinatorialpatternmatching

Step 1: Match Detection

• If x1…xn and y1…yn match with at most k mismatches, they

must share an l-tuple that is perfectly matched, where we have

l = n/(k + 1).

• Here j denotes the largest integer q with q ≤ j

• Where does this formula come from?

• Break string of length n into k+1 parts, each each of length

n/(k + 1)

• k mismatches can affect at most k of these k+1 parts.

• At least one of these k+1 parts is perfectly matched.

Page 92: Ch09 combinatorialpatternmatching

Step 1: Match Detection

1…l l +1…2l 2l +1…3l 3l +1…n

1 2 k k + 1

• Suppose k = 3. We would then have l = n / (k+1) =n/4:

• There are at most k mismatches in n, so at the very least there

must be one out of the k + 1 l-tuples without a mismatch.

Page 93: Ch09 combinatorialpatternmatching

Step 2: Match Verification

Query

Extend perfect

match of length l

until we find an

approximate

match of length n

with k

mismatches

Text

• For each l-match we find, try to extend the match further to see

if it is substantial.

Page 94: Ch09 combinatorialpatternmatching

Filtration: Changing l Changes Performance

l -tuple

length

Shorter perfect matches required

Performance decreases

n

2

n

n

3

n

4

n

5

n

6

k 5

k 4

k 3

k 2

k 1

k 0

Page 95: Ch09 combinatorialpatternmatching

Section 8:

Comparing a Sequence

Against a Database

Page 96: Ch09 combinatorialpatternmatching

Local alignment is too slow…

),(

),(

),(

0

max

1,1

1,

,1

,

jiji

jji

iji

ji

wvs

ws

vss

• Quadratic local alignment is

too slow while looking for

similarities between long

strings (e.g. the entire

GenBank database).

• Guaranteed to find the

optimal local alignment.

• Sets the standard for

sensitivity.

Page 97: Ch09 combinatorialpatternmatching

Local alignment is too slow…

),(

),(

),(

0

max

1,1

1,

,1

,

jiji

jji

iji

ji

wvs

ws

vss

• BLAST: Basic Local

Alignment Search Tool

• Created in 1990 by

Altschul, Gish, Miller,

Myers, & Lipman.

• Idea: Search sequence

databases for local

alignments to a query.

Page 98: Ch09 combinatorialpatternmatching

Features of BLAST

• Great improvement in speed, with a modest decrease in

sensitivity.

• Minimizes search space instead of exploring entire search

space between two sequences.

• Finds short exact matches (―seeds‖), and only explores locally

around these ―hits.‖

Page 99: Ch09 combinatorialpatternmatching

Section 9:

BLAST

Algorithm/Statistics

Page 100: Ch09 combinatorialpatternmatching

BLAST Algorithm

• Keyword search of all words of length w from a query of

length n in database of length m with score above threshold.

• w = 11 for DNA queries

• w =3 for proteins

• Perform local alignment extension for each keyword.

• Extend results until longest match above a given threshold is

achieved.

• Runtime: O(nm)

Page 101: Ch09 combinatorialpatternmatching

BLAST Algorithm

Query: 22 VLRDNIQGITKPAIRRLARRGGVKRISGLIYEETRGVLK 60

+++DN +G + IR L G+K I+ L+ E+ RG++K

Sbjct: 226 IIKDNGRGFSGKQIRNLNYGIGLKVIADLV-EKHRGIIK 263

Query: KRHRKVLRDNIQGITKPAIRRLARRGGVKRISGLIYEETRGVLKIFLENVIRD

keyword

GVK 18

GAK 16

GIK 16

GGK 14

GLK 13

GNK 12

GRK 11

GEK 11

GDK 11

neighborhood

score threshold

(T = 13)

Neighborhood

words

High-scoring Pair (HSP)

extension

Page 102: Ch09 combinatorialpatternmatching

Original BLAST

• Dictionary

• All words of length w

• Alignment

• Ungapped extensions until score falls below some

statistical threshold.

• Output

• All local alignments with score > threshold

Page 103: Ch09 combinatorialpatternmatching

Original BLAST: Example

From lectures by Serafim Batzoglou (Stanford)

• w = 4

• Exact keyword

match of GGTC

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

Page 104: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 105: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 106: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

• Extend diagonals

with mismatches

until score is under

50%.

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 107: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

• Extend diagonals

with mismatches

until score is under

50%.

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 108: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

• Extend diagonals

with mismatches

until score is under

50%.

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 109: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

• Extend diagonals

with mismatches

until score is under

50%.

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 110: Ch09 combinatorialpatternmatching

Original BLAST: Example

• w = 4

• Exact keyword

match of GGTC

• Extend diagonals

with mismatches

until score is under

50%.

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 111: Ch09 combinatorialpatternmatching

A C G A A G T A A G G T C C A G T

Original BLAST: Example

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

• w = 4

• Exact keyword

match of GGTC

• Extend diagonals

with mismatches

until score is under

50%.

• Output result:

• GTAAGGTCC

• GTTAGGTCC

From lectures by Serafim Batzoglou (Stanford)

Page 112: Ch09 combinatorialpatternmatching

Gapped BLAST: Example

• Original BLAST exact

keyword search, THEN:

• Extend with gaps

around ends of exact

match until score <

threshold

• Output result

GTAAGGTCCAGT

GTTAGGTC-AGT

A C G A A G T A A G G T C C A G T

G

A

T

C

C T

G

G

A

T

T

G

C

G

A

From lectures by Serafim Batzoglou (Stanford)

Page 113: Ch09 combinatorialpatternmatching

Incarnations of BLAST

• BLASTN: Nucleotide-nucleotide

• BLASTP: Protein-protein

• BLASTX: Translated query vs. protein database

• TBLASTN: Protein query vs. translated database

• TBLASTX: Translated query vs. translated database (6 frames

each)

Page 114: Ch09 combinatorialpatternmatching

Incarnations of BLAST

• PSI-BLAST: Find members of a protein family or build a

custom position-specific score matrix.

• MegaBLAST: Search longer sequences with fewer

differences.

• WU-BLAST (Wash U BLAST): Optimized, added features.

Page 115: Ch09 combinatorialpatternmatching

Assessing Sequence Similarity

• We need to know how strong an alignment can be expected to

be from chance alone.

• ―Chance‖ relates to comparison of sequences that are

generated randomly based upon a certain sequence model.

• Sequence models may take into account:

• G+C content

• Poly-A tails

• ―Junk‖ DNA

• Codon bias

• Etc.

Page 116: Ch09 combinatorialpatternmatching

BLAST: Segment Score

• BLAST uses scoring matrices (d) to improve on efficiency of

match detection.

• Some proteins may have very different amino acid sequences,

but are still similar.

• For any two l-mers x1…xl and y1…yl :

• Segment pair: Pair of l-mers, one from each sequence.

• Segment score:

d xi, y

i i1

l

Page 117: Ch09 combinatorialpatternmatching

BLAST: Locally Maximal Segment Pairs

• A segment pair is locally maximal if its score can‘t be

improved by extending or shortening.

• Statistically significant locally maximal segment pairs are of

biological interest.

• BLAST finds all locally maximal segment pairs with scores

above some threshold.

• A significantly high threshold will filter out some statistically

insignificant matches.

Page 118: Ch09 combinatorialpatternmatching

BLAST: Statistics

• Threshold: Altschul-Dembo-Karlin statistics.

• Identifies smallest segment score that is unlikely to happen by

chance.

• # matches above q has mean E(q) = Kmne-lq; K is a constant, m

and n are the lengths of the two compared sequences.

• Parameter l is positive root of S x,y in A(pxpyed(x,y)) = 1, where px

and py are frequencies of amino acids x and y, and A is the

twenty letter amino acid alphabet

Page 119: Ch09 combinatorialpatternmatching

P-values

• The probability of finding b HSPs with a score ≥S is given by:

• For b = 0, that chance is:

• Thus the probability of finding at least one HSP with a score ≥

S is:

(eEEb)

b!

e E

P 1 eE

Page 120: Ch09 combinatorialpatternmatching

Sample BLAST output

Score E

Sequences producing significant alignments: (bits) Value

gi|18858329|ref|NP_571095.1| ba1 globin [Danio rerio] >gi|147757... 171 3e-44

gi|18858331|ref|NP_571096.1| ba2 globin; SI:dZ118J2.3 [Danio rer... 170 7e-44

gi|37606100|emb|CAE48992.1| SI:bY187G17.6 (novel beta globin) [D... 170 7e-44

gi|31419195|gb|AAH53176.1| Ba1 protein [Danio rerio] 168 3e-43

ALIGNMENTS

>gi|18858329|ref|NP_571095.1| ba1 globin [Danio rerio]

Length = 148

Score = 171 bits (434), Expect = 3e-44

Identities = 76/148 (51%), Positives = 106/148 (71%), Gaps = 1/148 (0%)

Query: 1 MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTQRFFESFGDLSTPDAVMGNPK 60

MV T E++A+ LWGK+N+DE+G +AL R L+VYPWTQR+F +FG+LS+P A+MGNPK

Sbjct: 1 MVEWTDAERTAILGLWGKLNIDEIGPQALSRCLIVYPWTQRYFATFGNLSSPAAIMGNPK 60

Query: 61 VKAHGKKVLGAFSDGLAHLDNLKGTFATLSELHCDKLHVDPENFRLLGNVLVCVLAHHFG 120

V AHG+ V+G + ++DN+K T+A LS +H +KLHVDP+NFRLL + + A FG

Sbjct: 61 VAAHGRTVMGGLERAIKNMDNVKNTYAALSVMHSEKLHVDPDNFRLLADCITVCAAMKFG 120

• BLAST of human betaglobin protein against zebra fish:

Page 121: Ch09 combinatorialpatternmatching

Sample BLAST оutput

Score E

Sequences producing significant alignments: (bits) Value

gi|19849266|gb|AF487523.1| Homo sapiens gamma A hemoglobin (HBG1... 289 1e-75

gi|183868|gb|M11427.1|HUMHBG3E Human gamma-globin mRNA, 3' end 289 1e-75

gi|44887617|gb|AY534688.1| Homo sapiens A-gamma globin (HBG1) ge... 280 1e-72

gi|31726|emb|V00512.1|HSGGL1 Human messenger RNA for gamma-globin 260 1e-66

gi|38683401|ref|NR_001589.1| Homo sapiens hemoglobin, beta pseud... 151 7e-34

gi|18462073|gb|AF339400.1| Homo sapiens haplotype PB26 beta-glob... 149 3e-33

ALIGNMENTS

>gi|28380636|ref|NG_000007.3| Homo sapiens beta globin region (HBB@) on chromosome 11

Length = 81706

Score = 149 bits (75), Expect = 3e-33

Identities = 183/219 (83%)

Strand = Plus / Plus

Query: 267 ttgggagatgccacaaagcacctggatgatctcaagggcacctttgcccagctgagtgaa 326

|| ||| | || | || | |||||| ||||| ||||||||||| ||||||||

Sbjct: 54409 ttcggaaaagctgttatgctcacggatgacctcaaaggcacctttgctacactgagtgac 54468

• BLAST of human betaglobin protein against zebra fish:

Page 122: Ch09 combinatorialpatternmatching

Section 10:

PatternHunter and BLAT

Page 123: Ch09 combinatorialpatternmatching

Timeline

• 1970: Needleman-Wunsch global alignment algorithm

• 1981: Smith-Waterman local alignment algorithm

• 1985: FASTA

• 1990: BLAST (basic local alignment search tool)

• 2000s: Faster algorithms evolve for genome/genome:

• Pattern Hunter

• BLAT

Page 124: Ch09 combinatorialpatternmatching

BLAT vs. BLAST

• BLAT (BLAST-Like Alignment Tool): Same idea as

BLAST—locate short sequence hits and extend.

• Differences between BLAT and BLAST:

• BLAST builds an index of the query sequence and then

scans linearly through the database.

• BLAT builds an index of the database and scans linearly

through the query sequence.

• Index is stored in RAM, resulting in faster searches.

Page 125: Ch09 combinatorialpatternmatching

BLAT: Indexing

• An index is built that contains the positions of each k-mer in

the genome.

• Each k-mer in the query sequence is compared to each k-mer

in the index.

• A list of ‗hits‘ is generated – matching k-mer positions in

cDNA and in genome.

Page 126: Ch09 combinatorialpatternmatching

• Steps:

1. Break cDNA into 500 base chunks.

2. Use an index to find regions in genome similar to each

chunk of cDNA.

3. Construct alignment between genomic regions and cDNA

chunks.

4. Use dynamic programming to stitch together alignments

of chunks into the alignment of the whole.

BLAT: Fast cDNA Alignments

Page 127: Ch09 combinatorialpatternmatching

Indexing: An Example

Position of 3-mer in query, genome

• Here is an example with k = 3:Genome: cacaattatcacgaccgc

3-mers (non-overlapping): cac aat tat cac gac cgc

Index: aat 3 gac 12

cac 0,9 tat 6

cgc 15

cDNA (query sequence): aattctcac

3-mers (overlapping): aat att ttc tct ctc tca cac

0 1 2 3 4 5 6

Hits: aat 0,3

cac 6,0

cac 6,9

clump: cacAATtatCACgaccgc

Page 128: Ch09 combinatorialpatternmatching

However…

• BLAT was designed to find sequences of 95% and greater

similarity of length > 40.

• Therefore we may miss more divergent or shorter sequence

alignments.

Page 129: Ch09 combinatorialpatternmatching

PatternHunter: faster and even more sensitive

• BLAST: matches short

consecutive sequences

(consecutive seed)

• Length = k

• Example (k = 11):

11111111111

• Each 1 represents a

―match.‖

• PatternHunter: matches

short non-consecutive

sequences (spaced seed).

• Increases sensitivity by

locating homologies that

would otherwise be missed.

• Example (a spaced seed of

length 18 with 11 matches):

111010010100110111

• Each 0 represents a ―don‘t

care‖, so there can be a

match or a mismatch.

Page 130: Ch09 combinatorialpatternmatching

Spaced Seeds

• Example of a hit using a spaced seed:

• How does this result in better sensitivity?

Page 131: Ch09 combinatorialpatternmatching

Why is PH Better?

• BLAST: redundant hits

• This results in > 1 hit and creates clusters of redundant hits.

• Example:

• PatternHunter: very few

redundant hits.

• Example:

Page 132: Ch09 combinatorialpatternmatching

Why is PH Better?

GAGTACTCAACACCAACATTAGTGGGCAATGGAAAAT

|||||||||||||||||||||||||||||||||||||

GAATACTCAACAGCAACATCAATGGGCAGCAGAAAAT

• In this example, despite a clear similarity, there is no sequence

of continuous matches longer than length 9.

• BLAST uses a length 11 and because of this, BLAST does

not recognize this as a hit!

• Resolving this would require reducing the seed length to 9,

which would have a damaging effect on speed.

• Example:

9 matches

Page 133: Ch09 combinatorialpatternmatching

Advantage of Gapped Seeds

11 positions

11 positions

10 positions

Page 134: Ch09 combinatorialpatternmatching

Why is PH Better?

1. Higher hit probability.

2. Lower expected number of random hits.

Page 135: Ch09 combinatorialpatternmatching

Use of Multiple Seeds

• Basic Search Algorithm:

1. Select a group of spaced seed models.

2. For each hit of each model, conduct extension to find a

homology.

Page 136: Ch09 combinatorialpatternmatching

PatternHunter and BLAT vs. BLAST

• PatternHunter is 5-100 times faster than BLASTN, depending

on data size, at the same sensitivity.

• BLAT is several times faster than BLAST, but best results are

limited to closely related sequences.