monday, november 26, 12 - archive.apachecon.comarchive.apachecon.com/eu2012/presentations/08... ·...

Monday, November 26, 12

The CouchDB Implementation


Jan [email protected]@janlCouchDB PMC ChairCommitter #2


Thanks for the inviteGlad to be here

JSConf EU / BBuzz / JSFAB

Any Database


- FS integration / raw storage - core data structures - core features - API

FILE SYSTEM

FILE SYSTEM ACCESS

CORE DATA STRUCTURES

CORE FEATURES

API


CouchDB is no different


- couch_file - couch_btree - couch_db / couch_doc / couch_mr / couch_replicator / etcpp - couch_httpd*

FILE SYSTEM

COUCH_FILE

COUCH_BTREE

COUCH_DB +

COUCH_HTTP*

COUCH_DOCCOUCH_MRCOUCH_REPL…


Core Datastructures


- 2 x b+tree & data interleaved - append only, mvcc - full fsync control

- Can answer: - Data for $key - What happened $since

- Used for core data storage - As well as indexes

- Everything else is built on top

Behold theb+tree


by-id


A B C D E F G H I J K L …


A B C D E F

DO

C_G

H I J K L …


A B C

DO

C_D

E F

DO

C_G

H I J K L …


A B C

DO

C_D

E F

DO

C_G

H I J

DO

C_K

L …


DO

C_D

DO

C_G

DO

C_K


by-seqor

“what happened since?”


1. D

OC_G

2. D

OC_D

3. D

OC_K


The CouchDBFile Format






HEA

DER






DO

C_A

HEA

DER






DO

C_A

BY

ID

ID

X A

HEA

DER






DO

C_A

BY

ID

ID

X A

BY

SEQ

ID

X A

HEA

DER






DO

C_A

BY

ID

ID

X A

BY

SEQ

ID

X A

HEA

DER

FOO

TER






DO

C_A

DO

C_B

BY

ID

ID

X A

BY

SEQ

ID

X A

HEA

DER

FOO

TER






DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

HEA

DER

FOO

TER






DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

HEA

DER

FOO

TER






DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

HEA

DER

FOO

TER

FOO

TER







Bulk add + Delete

DO

C_A


Bulk add + Delete

DO

C_A

DO

C_B


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

FOO

TER


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

FOO

TER

DEL

DO

C_A


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

FOO

TER

DEL

DO

C_A

BY

ID

ID

X A


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

FOO

TER

DEL

DO

C_A

BY

ID

ID

X A

BY

SEQ

ID

X A


Bulk add + Delete

DO

C_A

DO

C_B

BY

ID

ID

X A

BY

ID

ID

X B

BY

SEQ

ID

X A

BY

SEQ

ID

X B

FOO

TER

DEL

DO

C_A

BY

ID

ID

X A

BY

SEQ

ID

X A

FOO

TER


Bulk add + Delete

Operational Consequences


- efficient on spinning disk, “tape” - btree = wide, upper layers in disk cache - backup with cp $a $b

- compaction hurts

Core Features (using by-seq)


- Replication - Indexing / Views / GeoCouch / Lucene / ES etc. - /_changes - Compaction


Replication

DATABASE AMonday, November 26, 12

Replication

DATABASE A

1


Replication

DATABASE A

2

1


Replication

3

DATABASE A

2

1


Replication

3

DATABASE A

2

1

4


Replication

3

DATABASE A

2

1

4

DATABASE BMonday, November 26, 12

Replication

3

DATABASE A

2

1

4

DATABASE B

1


Replication

3

DATABASE A

2

1

4

DATABASE B

2

1


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

5


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

6

5


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

7

6

5


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

7

6

5

8


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

7

6

5

8

5


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

7

6

5

8

6

5


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

7

6

5

8

7

6

5


Replication

3

DATABASE A

2

1

4

3

DATABASE B

2

1

4

7

6

5

8

7

6

5

8


Replication


Indexing

DATABASE A

1


Indexing

DATABASE A

2

1


Indexing

3

DATABASE A

2

1


Indexing

3

DATABASE A

2

1

4


Indexing

3

DATABASE A

2

1

4

INDEX AMonday, November 26, 12

Indexing

3

DATABASE A

2

1

4

INDEX A

1


Indexing

3

DATABASE A

2

1

4

INDEX A

2

1


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

5


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

6

5


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

7

6

5


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

7

6

5

8


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

7

6

5

8

5


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

7

6

5

8

6

5


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

7

6

5

8

7

6

5


Indexing

3

DATABASE A

2

1

4

3

INDEX A

2

1

4

7

6

5

8

7

6

5

8


Indexing


/_changes

DATABASE A

1


/_changes

DATABASE A

2

1


/_changes

3

DATABASE A

2

1


/_changes

3

DATABASE A

2

1

4


/_changes

3

DATABASE A

2

1

4

5


/_changes

3

DATABASE A

2

1

4

6

5


/_changes

3

DATABASE A

2

1

4

7

6

5


/_changes

3

DATABASE A

2

1

4

7

6

5

8


/_changes


Compaction

DATABASE A

1. DOC_A


Compaction

DATABASE A

2. DOC_B

1. DOC_A


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

5. DOC_D


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

6. DOC_B

5. DOC_D


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

COMPACT AMonday, November 26, 12

Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

3. DOC_C

COMPACT AMonday, November 26, 12

Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

3. DOC_C

COMPACT A

4. DOC_A


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

3. DOC_C

COMPACT A

4. DOC_A

5. DOC_D


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

3. DOC_C

COMPACT A

4. DOC_A

6. DOC_B

5. DOC_D


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

3. DOC_C

COMPACT A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D


Compaction

3. DOC_C

DATABASE A

2. DOC_B

1. DOC_A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G

3. DOC_C

COMPACT A

4. DOC_A

7. DOC_F

6. DOC_B

5. DOC_D

8. DOC_G


Compaction

Erlang


- Small codebase - Efficient in small teams

- Supervision tree - Isolated processes - Concurrency

- Portable runtime

- Hard to recruit for - Steep ramp-on - Bit of an operational black box (nine nines story) - Bit hard as an install-dependency

COUCH_SERVER

COUCH_HTTP

COUCH_QUERY_SERVER

COUCH_LOG

COUCH_DB

COUCH_DB_UPDATER

COUCH_FILE


- Small codebase - Efficient in small teams

- Supervision tree - Isolated processes - Concurrency

- Portable runtime

- Hard to recruit for - Steep ramp-on - Bit of an operational black box (nine nines story) - Bit hard as an install-dependency

Potential Improvements


- Smarter compactor - Smarter file-storage - Less custom HTTP handling - More indexers

The End


Thanks!


monday, november 26, 12 - archive.apachecon.comarchive.apachecon.com/eu2012/presentations/08... ·...

Documents