nosql @ qcon sp
DESCRIPTION
TRANSCRIPT
![Page 1: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/1.jpg)
noSQL
quarta-feira, 8 de setembro de 2010
![Page 2: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/2.jpg)
. hype
quarta-feira, 8 de setembro de 2010
![Page 3: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/3.jpg)
história...
quarta-feira, 8 de setembro de 2010
![Page 4: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/4.jpg)
• Hierarchical (IMS): late 1960’s and 1970’s • Directed graph (CODASYL): 1970’s • Relational: 1970’s and early 1980’s • Entity-Relationship: 1970’s • Extended Relational: 1980’s • Semantic: late 1970’s and 1980’s• Object-oriented: late 1980’s and early 1990’s • Object-relational: late 1980’s and early 1990’s • Semi-structured (XML): late 1990’s to late 2000’s• The next big thing: ???
ref: What Goes Around Comes Around por Michael Stonebraker e Joey Hellerstein
modelos
quarta-feira, 8 de setembro de 2010
![Page 5: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/5.jpg)
next big thing?
quarta-feira, 8 de setembro de 2010
![Page 6: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/6.jpg)
definição...
quarta-feira, 8 de setembro de 2010
![Page 7: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/7.jpg)
abaixo ao banco de
dados relacional!
quarta-feira, 8 de setembro de 2010
![Page 8: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/8.jpg)
abaixo ao banco de dados relacional!
como bala de prata!
quarta-feira, 8 de setembro de 2010
![Page 9: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/9.jpg)
momento histórico...
quarta-feira, 8 de setembro de 2010
![Page 10: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/10.jpg)
quarta-feira, 8 de setembro de 2010
![Page 11: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/11.jpg)
resolver problemas específicos
quarta-feira, 8 de setembro de 2010
![Page 12: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/12.jpg)
quais problemas?
quarta-feira, 8 de setembro de 2010
![Page 13: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/13.jpg)
Architectural Anti PatternsNotes on Data Distribution and Handling Failures
quarta-feira, 8 de setembro de 2010
![Page 14: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/14.jpg)
Required Listening: Frank Zappa - One size fits all
quarta-feira, 8 de setembro de 2010
![Page 15: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/15.jpg)
Anti Patterns
• Evolution from SQL Anti Patterns (NoSQL:br May 2010)• More than just RDBMS• Large volumes of data• Distribution• Architecture• Research on other tools• Message Queues, DHT, Job Schedulers, NoSQL• Indexing, Map/Reduce
quarta-feira, 8 de setembro de 2010
![Page 16: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/16.jpg)
RDBMS Anti Patterns Not all things fit on a relational database, single ou distributed
• The eternal table-as-a-tree • Dynamic table creation• Table as cache • Table as queue • Table as log file• Stoned Procedures• Row Alignment• Extreme JOINs• Your scheme must be printed in an A3 sheet.• Your ORM issue full queries for Dataset iterations
quarta-feira, 8 de setembro de 2010
![Page 17: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/17.jpg)
Doing it wrong, Junior !
quarta-feira, 8 de setembro de 2010
![Page 18: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/18.jpg)
The eternal treeProblem: Most threaded discussion example uses something like a table which contains all threads and answers, relating to each other by an id. Usually the developer will come up with his own binary-tree version to manage this mess.
id - parent_id -author - text1 - 0 - gleicon - hello world2 - 1 - elvis - shout !
Alternative: Document storage:{ thread_id:1, title: 'the meeting', author: 'gleicon', replies:[ { 'author': elvis, text:'shout', replies:[{...}] } ]}
quarta-feira, 8 de setembro de 2010
![Page 19: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/19.jpg)
Dynamic table creationProblem: To avoid huge tables, one must come with a "dynamic schema". For example, lets think about a document management company, which is adding new facilities over the country. For each storage facility, a new table is created:
item_id - row - column - stuff1 - 10 - 20 - cat food2 - 12 - 32 - trout
Now you have to come up with "dynamic queries", which will probably query a "central storage" table and issue a huge join to check if you have enough cat food over the country.
Alternatives: - Document storage, modeling a facility as a document- Key/Value, modeling each facility as a SET
quarta-feira, 8 de setembro de 2010
![Page 20: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/20.jpg)
Table as cacheProblem: Complex queries demand that a result be stored in a separated table, so it can be queried quickly. Worst than views
Alternatives: - Really ?
- Memcached
- Redis + AOF + EXPIRE
- De-normalization
quarta-feira, 8 de setembro de 2010
![Page 21: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/21.jpg)
Table as queueProblem: A table which holds messages to be completed. Worse, they must be ordered bytime of creation.
Corolary: Job Scheduler table
Alternatives: - RestMQ, Resque
- Any other message broker
- Redis (LISTS - LPUSH + RPOP)
- Use the right tool
quarta-feira, 8 de setembro de 2010
![Page 22: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/22.jpg)
Table as log fileProblem: A table in which data gets written as a log file. From time to time it needs to be purged. Truncating this table once a day usually is the first task assigned to new DBAs.
Alternative:
- MongoDB capped collection
- Redis, and RRD pattern
- RIAK
quarta-feira, 8 de setembro de 2010
![Page 23: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/23.jpg)
Stoned proceduresProblem: Stored procedures hold most of your applications logic. Also, some triggers are used to - well - trigger important data events.
SP and triggers has the magic property of vanishing of our memories and being impossible to keep versioned.
Alternative: - Now be careful so you dont use map/reduce as modern stoned procedures. Unfit for real time search/processing
- Use your preferred language for business stuff, and let event handling to pub/sub or message queues.
quarta-feira, 8 de setembro de 2010
![Page 24: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/24.jpg)
Row AlignmentProblem: Extra rows are created but not used, just in case. Usually they are named as a1, a2, a3, a4 and called padding.
There's good will behind that, specially when version 1 of the software needed an extra column in a 150M lines database and it took 2 days to run an ALTER TABLE. But that's no excuse.
Alternative:
- Quit being cheap. Quit feeling 'hacker' about padding
- Document based databases as MongoDB and CouchDB, has no schema. New atributes are local to the document and can be added easily.
quarta-feira, 8 de setembro de 2010
![Page 25: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/25.jpg)
Extreme JOINsProblem: Business stuff modeled as tables. Table inheritance (Product -> SubProduct_A). To find the complete data for a user plan, one must issue gigantic queries with lots of JOINs.
Alternative:
- Document storage, as MongoDB might help having important information together.
- De-normalization
- Serialized objects
quarta-feira, 8 de setembro de 2010
![Page 26: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/26.jpg)
Your scheme fits in an A3 sheetProblem: Huge data schemes are difficult to manage. Extreme specialization creates tables which converges to key/value model. The normal form get priority over common sense.
Product_A Product_Bid - desc id - desc
Alternatives: - De-normalization- Another scheme ? - Document store for flattening model- Key/Value- See 'Extreme JOINs'
quarta-feira, 8 de setembro de 2010
![Page 27: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/27.jpg)
Your ORM ...Problem: Your ORM issue full queries for dataset iterations, your ORM maps and creates tables which mimics your classes, even the inheritance, and the performance is bad because the queries are huge, etc, etc
Alternative:
- Apart from denormalization and good old common sense, ORMs are trying to bridge two things with distinct impedance.
- There is nothing to relational models which maps cleanly to classes and objects. Not even the basic unit which is the domain(set) of each column. Black Magic ?
quarta-feira, 8 de setembro de 2010
![Page 28: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/28.jpg)
No silver bullet
- Think about data handling and your system architecture
- Think outside the norm
- De-normalize
- Simplify
- Know stuff (Message queues, NoSQL, DHT)
quarta-feira, 8 de setembro de 2010
![Page 29: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/29.jpg)
Cycle of changes - Product A
1.There was the database model2.Then, the cache was needed. Performance was no good.3.Cache key: query, value: resultset4.High or inexistent expiration time [w00t]
(Now there's a turning point. Data didn't need to change often. Denormalization was a given with cache)
5. The cache needs to be warmed or the app wont work.6. Key/Value storage was a natural choice. No data on MySQL anymore.
quarta-feira, 8 de setembro de 2010
![Page 30: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/30.jpg)
Cycle of changes - Product B
1.Postgres DB storing crawler results.2.There was a counter in each row, and updating this counter
caused contention errors.3.Memcache for reads. Performance is better.4.First MongoDB test, no more deadlocks from counter update.5.Data model was simplified, the entire crawled doc was
stored.
quarta-feira, 8 de setembro de 2010
![Page 31: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/31.jpg)
Stuff to think about
Think if the data you use aren't de-normalized somewhere (cached)
Most of the anti-patterns signals that there are architectural issues instead of only database issues.
The NoSQL route (or at least a partial NoSQL route) may simplify it.
Are you dependent on cache ? Does your application fails when there is no cache ? Does it just slows down ?
Think about the way to put and to get back your data from the database (be it SQL or NoSQL).
quarta-feira, 8 de setembro de 2010
![Page 32: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/32.jpg)
arquiteturaquarta-feira, 8 de setembro de 2010
![Page 33: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/33.jpg)
armazenamento de dados NÃO
tem sido [a muito tempo]
considerado parte de
arquitetura
quarta-feira, 8 de setembro de 2010
![Page 34: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/34.jpg)
cada escolha uma
renúncia
quarta-feira, 8 de setembro de 2010
![Page 35: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/35.jpg)
padrões
quarta-feira, 8 de setembro de 2010
![Page 36: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/36.jpg)
how-to
quarta-feira, 8 de setembro de 2010
![Page 37: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/37.jpg)
quarta-feira, 8 de setembro de 2010
![Page 38: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/38.jpg)
acid
quarta-feira, 8 de setembro de 2010
![Page 39: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/39.jpg)
(quarta-feira, 8 de setembro de 2010
![Page 40: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/40.jpg)
existe nosqlacid
quarta-feira, 8 de setembro de 2010
![Page 41: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/41.jpg)
)quarta-feira, 8 de setembro de 2010
![Page 42: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/42.jpg)
ref: The CAP Theorem por Seth Gilbert & Nancy Lynch
CAP
quarta-feira, 8 de setembro de 2010
![Page 43: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/43.jpg)
CAP
onsistencyvailabilityartition Tolerance
quarta-feira, 8 de setembro de 2010
![Page 44: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/44.jpg)
quarta-feira, 8 de setembro de 2010
![Page 45: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/45.jpg)
BASE
ref: BASE: an Acid Alternative por Dan Pritchettquarta-feira, 8 de setembro de 2010
![Page 46: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/46.jpg)
BASE
asicallyvailableoft Stateeventually Consistent
quarta-feira, 8 de setembro de 2010
![Page 47: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/47.jpg)
Eventually Consistency
ref: Eventually Consistent por Werner Vogelsquarta-feira, 8 de setembro de 2010
![Page 48: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/48.jpg)
eventual em português: pode ou não ocorrer
eventual em inglês: irá ocorrer em algum
momento
quarta-feira, 8 de setembro de 2010
![Page 49: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/49.jpg)
Consitência em Momento
Indeterminado
@mdedianaquarta-feira, 8 de setembro de 2010
![Page 50: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/50.jpg)
consistência
W+R > N
quarta-feira, 8 de setembro de 2010
![Page 51: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/51.jpg)
durabilidade
ref: The End of an Architectural Era por Michael Stonebraker & al.quarta-feira, 8 de setembro de 2010
![Page 52: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/52.jpg)
★ latência★ performance★ particionamento★ distribuição★ replicação
ainda tem...
quarta-feira, 8 de setembro de 2010
![Page 53: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/53.jpg)
lembre-se
vc não está criando uma solução de escala
intergaláctica com tolerância a falhas aleatórias
entre datacenters espalhados em diversas
localizações geográficas e outras dimensões
quarta-feira, 8 de setembro de 2010
![Page 54: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/54.jpg)
sacou a importância
da arquitetura?
quarta-feira, 8 de setembro de 2010
![Page 55: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/55.jpg)
com tantas definições...com tantos conceitos...com tantos tradeoffs...
com tantos....
quarta-feira, 8 de setembro de 2010
![Page 56: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/56.jpg)
como o nosql se tornou tão
sexy e popular?
quarta-feira, 8 de setembro de 2010
![Page 57: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/57.jpg)
apesar de tudo....
é fácil usar!quarta-feira, 8 de setembro de 2010
![Page 58: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/58.jpg)
persitência poliglota
quarta-feira, 8 de setembro de 2010
![Page 59: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/59.jpg)
Perguntas?
quarta-feira, 8 de setembro de 2010
![Page 60: noSQL @ QCon SP](https://reader034.vdocuments.site/reader034/viewer/2022042613/547e02c75906b5d3718b45fb/html5/thumbnails/60.jpg)
@porcelli
linkedin.com/in/alexandreporcelli
porcelli.com.br
Obrigado
github.com/porcelli
@gleicon
linkedin.com/in/gleicon
zenmachine.wordpress.com
github.com/gleicon
quarta-feira, 8 de setembro de 2010