fractal composition of meaning: toward a collage theorem for language simon d. levy department of...

Fractal Composition of Meaning:Fractal Composition of Meaning:Toward a Collage Theorem for Toward a Collage Theorem for LanguageLanguage

Simon D. Levy Simon D. Levy

Department of Computer ScienceDepartment of Computer Science

Washington and Lee UniversityWashington and Lee University

Lexington, VA 24450Lexington, VA 24450

http://www.cs.wlu.edu/~levyhttp://www.cs.wlu.edu/~levy

Part I: Self-SimilarityPart I: Self-Similarity

... But I know, too,

That

the blackbird is involved

in

what what

... But I know, too,

That the blackbird is involved

In what

– Wallace Stevens

I know.

Part II: A Little MathPart II: A Little Math

Iterated Function SystemsIterated Function Systems

• IFS: Another way to make a fractal

• Start with an arbitrary initial image

• Apply a set of contractive affine transforms

• Repeat until image no longer changes

• E.g., Sierpinski Triangle...

A02

T i ,i 1,2,. .. ,n

Sierpinski TriangleSierpinski Triangle

T 1xy

.5 00 .5

xy

00

T 2xy

.5 00 .5

xy

0.5

T 3xy

.5 00 .5

xy

.5

.5

I.e., three half-size copies, one in each of three corners of [0,1] 2

Sierpinski Triangle: 0 IterationsSierpinski Triangle: 0 Iterations

Sierpinski Triangle: 1 IterationSierpinski Triangle: 1 Iteration

Sierpinski Triangle: 2 IterationsSierpinski Triangle: 2 Iterations

Sierpinski Triangle: 3 IterationsSierpinski Triangle: 3 Iterations

Sierpinski Triangle: 4 IterationsSierpinski Triangle: 4 Iterations

Sierpinski Triangle: 5 IterationsSierpinski Triangle: 5 Iterations

Sierpinski Triangle: 6 IterationsSierpinski Triangle: 6 Iterations

Sierpinski Triangle: 7 IterationsSierpinski Triangle: 7 Iterations

Sierpinski Triangle: New Initial Sierpinski Triangle: New Initial ImageImage

Sierpinski TriangleSierpinski Triangle

Sierpinski TriangleSierpinski Triangle

Sierpinski TriangleSierpinski Triangle

Sierpinski TriangleSierpinski Triangle

Sierpinski TriangleSierpinski Triangle

Sierpinski TriangleSierpinski Triangle

Sierpinski TriangleSierpinski Triangle

IFS Fractals in NatureIFS Fractals in Nature

Fractal Image CompressionFractal Image Compression

• Doesn't matter what image we start with

• All information needed to represent final “target”

image is contained in transforms

• Instead of storing millions of pixels, determine

transforms for target image, and store them

• How to determine transforms?

T i

A0

The Collage TheoremThe Collage Theorem

• Let

• is the attractor or “fixed point” of

• Collage Theorem (Barnsley 1988): Given

arbitrary target image , transforms encoding

are s.t.

• Use various practical methods to find

f A0iT i A0

limn

f n A0 f

A A

T i f AiT i A A

T i

Practical Fractal Image Practical Fractal Image CompressionCompression

• Most real-world images are only partially self-similar

• Arbitrary images can be partitioned into “tiles”,

each associated with a transform.

• Compression algorithm computes and stores

locations and transforms of tiles

Practical Fractal Image Practical Fractal Image CompressionCompression

Practical Fractal Image Practical Fractal Image CompressionCompression

Part III: UnificationPart III: Unification

The “Two Cultures”The “Two Cultures”

Discrete Symbols

Semantic Relations

Grammar Rules

Graph StructuresContinuous

Vectors

Metric Spaces

Continuous TransformsImages

Linguistics AI

Logic

Dynamical Systems

ChaosElectrical Engineering

Meanings as VectorsMeanings as Vectors“You shall know a word by the company it keeps”

– J. R. Firth

• Vector representation of a word encodes co-occurrence with other words

• Latent Semantic Analysis (Indexing) –

Singular Value Decomposition of co-

occurrence matrix on text; 300-dimensional

vectors [Landauer, Dumais, Kintsch]

Meanings as VectorsMeanings as Vectors

• Self-Organizing Maps – Collapse high-dimensional descriptions (binary features or real-val vectors) into 2-D [Kohonen]

• Simple Recurrent Networks – Hidden-

variable temporal model predicting next word

based on current; 150-D vectors [Elman]

Meanings as VectorsMeanings as Vectors

Fred says the woman arrived.The woman says fred arrived.Fred loves the woman.The woman loves fred.The woman arrived.Fred arrived.

Composing Meaningful Vectors Composing Meaningful Vectors

the woman.

Composing Meaningful Vectors Composing Meaningful Vectors

loves the woman.

Composing Meaningful Vectors Composing Meaningful Vectors

Fred loves the woman.

Part IV: ConclusionsPart IV: Conclusions

Advantages of Vector RepresentationsAdvantages of Vector Representations

• Meaning as a gradient phenomenon (semantic spaces = vector spaces)

• Can represent all transforms with a single

hidden-variable non-linear equation (“grammar”)

• Gradient-descent methods as learning model

• Principled, biologically plausible alternative to

“Words and Rules” approach [Chomsky, Pinker,

Fodor]

A Collage Theorem for LanguageA Collage Theorem for Language

A Collage Conjecture for LanguageA Collage Conjecture for Language

A Collage Conjecture for LanguageA Collage Conjecture for Language

A Collage Hypothesis for LanguageA Collage Hypothesis for Language

A Collage Hypothesis for LanguageA Collage Hypothesis for Language

A Collage S.W.A.G. for LanguageA Collage S.W.A.G. for Language

• Words/meanings are co-occurrence vectors.

• Compositions of meanings are transients to

words.

• “Correct” set of transients is one for which

word vectors form a subset of the attractor.