unsupervised domain adaptation: from practice to theory john blitzer texpoint fonts used in emf....
TRANSCRIPT
![Page 1: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/1.jpg)
Unsupervised Domain Adaptation: From Practice to Theory
John Blitzer
![Page 2: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/2.jpg)
...
...
??
?
Unsupervised Domain Adaptation
Running with Scissors
Title: Horrible book, horrible.
This book was horrible. I read half,
suffering from a headache the entire
time, and eventually i lit it on fire. 1 less
copy in the world. Don't waste your
money. I wish i had the time spent
reading this book back. It wasted my
life
...
...
Avante Deep Fryer; Black
Title: lid does not work well...
I love the way the Tefal deep fryer
cooks, however, I am returning my
second one due to a defective lid
closure. The lid may close initially,
but after a few uses it no longer
stays closed. I won’t be buying this
one again.
Source Target
![Page 3: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/3.jpg)
...
...
??
?
Target-Specific Features
Running with Scissors
Title: Horrible book, horrible.
This book was horrible. I read half,
suffering from a headache the entire
time, and eventually i lit it on fire. 1 less
copy in the world. Don't waste your
money. I wish i had the time spent
reading this book back. It wasted my
life
...
...
Avante Deep Fryer; Black
Title: lid does not work well...
I love the way the Tefal deep fryer
cooks, however, I am returning my
second one due to a defective lid
closure. The lid may close initially,
but after a few uses it no longer
stays closed. I won’t be buying this
one again.
...
...
Source Target
![Page 4: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/4.jpg)
Learning Shared Representations
fascinatingboring
read halfcouldn’t put it down
defectivesturdyleakinglike a charm
fantastichighly recommended
waste of moneyhorrible
Sourc
e
Target
![Page 5: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/5.jpg)
Shared Representations: A Quick Review
Blitzer et al. (2006, 2007). Shared CCA.Tasks: Part of speech tagging, sentiment.
Xue et al. (2008). Probabilistic LSA Task: Cross-lingual document classification.
Guo et al. (2009). Latent Dirichlet AllocationTask: Named entity recognition
Huang et al. (2009). Hidden Markov Models Task: Part of Speech Tagging
![Page 6: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/6.jpg)
What do you mean, theory?
![Page 7: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/7.jpg)
What do you mean, theory?
Statistical Learning Theory:
![Page 8: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/8.jpg)
What do you mean, theory?
Statistical Learning Theory:
![Page 9: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/9.jpg)
What do you mean, theory?
Classical Learning Theory:
![Page 10: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/10.jpg)
What do you mean, theory?
Adaptation Learning Theory:
![Page 11: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/11.jpg)
Goals for Domain Adaptation Theory
1. A computable (source) sample bound on target error
2. A formal description of empirical phenomena• Why do shared representations algorithms work?
3. Suggestions for future research
![Page 12: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/12.jpg)
Talk Outline
1. Target Generalization Bounds using Discrepancy Distance
[BBCKPW 2009]
[Mansour et al. 2009]
2. Coupled Subspace Learning
[BFK 2010]
![Page 13: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/13.jpg)
Formalizing Domain Adaptation
Source labeled data
Source distribution Target distribution
Target unlabeled data
![Page 14: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/14.jpg)
Formalizing Domain Adaptation
Source labeled data
Source distribution Target distribution
Semi-supervised adaptation
Target unlabeled data
Some target labels
![Page 15: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/15.jpg)
Formalizing Domain Adaptation
Source labeled data
Source distribution Target distribution
Semi-supervised adaptation
Target unlabeled data
Not in this talk
![Page 16: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/16.jpg)
A Generalization Bound
![Page 17: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/17.jpg)
A new adaptation bound
Bound from [MMR09]
![Page 18: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/18.jpg)
Discrepancy Distance
When good source models go bad
![Page 19: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/19.jpg)
Binary Hypothesis Error Regions
+
![Page 20: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/20.jpg)
Binary Hypothesis Error Regions
+
+
![Page 21: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/21.jpg)
Binary Hypothesis Error Regions
+
+
![Page 22: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/22.jpg)
Binary Hypothesis Error Regions
![Page 23: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/23.jpg)
Discrepancy Distance
lowlow high
When good source models go bad
![Page 24: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/24.jpg)
Computing Discrepancy Distance
Learn pairs of hypotheses to discriminate source from target
![Page 25: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/25.jpg)
Computing Discrepancy Distance
Learn pairs of hypotheses to discriminate source from target
![Page 26: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/26.jpg)
Computing Discrepancy Distance
Learn pairs of hypotheses to discriminate source from target
![Page 27: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/27.jpg)
Hypothesis Classes & Representations
Linear Hypothesis Class:
Induced classes from projections
3
0
1
001
...
Goals for
1)
2)
![Page 28: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/28.jpg)
A Proxy for the Best Model
Linear Hypothesis Class:
Induced classes from projections
3
0
1
001
...
Goals for
1)
2)
![Page 29: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/29.jpg)
Problems with the Proxy
3
0
1
00
...
1) 2)
![Page 30: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/30.jpg)
Goals
1. A computable bound
2. Description of shared representations
3. Suggestions for future research
![Page 31: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/31.jpg)
Talk Outline
1. Target Generalization Bounds using Discrepancy Distance
[BBCKPW 2009]
[Mansour et al. 2009]
2. Coupled Subspace Learning
[BFK 2010]
![Page 32: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/32.jpg)
Assumption: Single Linear Predictor
target-specific can’t be estimated from source alone. . . yet
source-specific target-specificshared
![Page 33: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/33.jpg)
Visualizing Single Linear Predictorfa
scin
atin
g
works well
don’t buy
source
target
shared
![Page 34: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/34.jpg)
Visualizing Single Linear Predictorfa
scin
atin
g
works well
don’t buy
![Page 35: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/35.jpg)
Visualizing Single Linear Predictorfa
scin
atin
g
works well
don’t buy
![Page 36: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/36.jpg)
Visualizing Single Linear Predictorfa
scin
atin
g
works well
don’t buy
![Page 37: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/37.jpg)
Dimensionality Reduction Assumption
![Page 38: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/38.jpg)
Visualizing Dimensionality Reductionfa
scin
atin
g
works well
don’t buy
![Page 39: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/39.jpg)
Visualizing Dimensionality Reductionfa
scin
atin
g
works well
don’t buy
![Page 40: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/40.jpg)
Representation Soundnessfa
scin
atin
g
works well
don’t buy
![Page 41: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/41.jpg)
Representation Soundnessfa
scin
atin
g
works well
don’t buy
![Page 42: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/42.jpg)
Representation Soundnessfa
scin
atin
g
works well
don’t buy
![Page 43: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/43.jpg)
Perfect Adaptationfa
scin
atin
g
works well
don’t buy
![Page 44: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/44.jpg)
Algorithm
![Page 45: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/45.jpg)
Generalization
![Page 46: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/46.jpg)
Generalization
![Page 47: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/47.jpg)
Canonical Correlation Analysis (CCA) [Hotelling 1935]
1) Divide feature space into disjoint views
Do not buy the Shark portable steamer. The trigger mechanism is defective.
not buy
trigger
defective
mechanism
2) Find maximally correlating projections
not buy
defectivetrigger
mechanism
![Page 48: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/48.jpg)
Canonical Correlation Analysis (CCA) [Hotelling 1935]
1) Divide feature space into disjoint views
Do not buy the Shark portable steamer. The trigger mechanism is defective.
not buy
trigger
defective
mechanism
2) Find maximally correlating projections
not buy
defectivetrigger
mechanism
Ando and Zhang (ACL 2005)
Kakade and Foster (COLT 2006)
![Page 49: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/49.jpg)
Square Loss: Kitchen Appliances
Books DVDs Electronics1.1
1.3
1.5
1.7
Naïve
Source Domain
Squa
re lo
ss (
s)
![Page 50: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/50.jpg)
Square Loss: Kitchen Appliances
Books DVDs Electronics1.1
1.3
1.5
1.7
Naïve
Coupled
In Domain
Source Domain
Squa
re lo
ss (
s)
![Page 51: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/51.jpg)
Using Target-Specific Features
mush bad quality warranty evenly
super easy great product
dishwasher
books kitchen
trite
the publisher
the author introduction to
illustrations
good reference
kitchen bookscritique
![Page 52: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/52.jpg)
Comparing Discrepancy & Coupled Bounds
Target: DVDs
Squa
re L
oss
Source Instances
true error
coupled bound
![Page 53: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/53.jpg)
Comparing Discrepancy & Coupled Bounds
Target: DVDs
Squa
re L
oss
Source Instances
![Page 54: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/54.jpg)
Comparing Discrepancy & Coupled Bounds
Target: DVDs
Squa
re L
oss
Source Instances
![Page 55: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/55.jpg)
Idea: Active Learning
Piyush Rai et al. (2010)
![Page 56: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/56.jpg)
Goals
1. A computable bound
2. Description of shared representations
3. Suggestions for future research
![Page 57: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/57.jpg)
Conclusion
1. Theory can help us understand domain adaptation better
2. Good theory suggests new directions for future research
3. There’s still a lot left to do• Connecting supervised and unsupervised adaptation• Unsupervised adaptation for problems with structure
![Page 58: Unsupervised Domain Adaptation: From Practice to Theory John Blitzer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.:](https://reader038.vdocuments.site/reader038/viewer/2022110400/56649dd05503460f94ac5596/html5/thumbnails/58.jpg)
Thanks
Collaborators
Shai Ben-DavidKoby CrammerDean FosterSham Kakade
Alex KuleszaFernando PereiraJenn Wortman
References
Ben-David et al. A Theory of Learning from Different Domains. Machine Learning 2009.
Mansour et al. Domain Adaptation: Learning Bounds and Algorithms. COLT 2009.