a27 vectorwise performance considerations_implementation_best_practices
TRANSCRIPT
![Page 1: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/1.jpg)
Vectorwise Implementation best practices
Mark Van de Wiel
Thursday, November 01, 2012
Director Product Management, Vectorwise
1 of 9 1 of 9 1 of 9 1 of 9 1 of 9 Confidential © 2012 Actian Corporation
![Page 2: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/2.jpg)
Agenda
2
Hardware
Operating system
Database configuration
Database design
Data loading
High availability
Monitoring
Confidential © 2012 Actian Corporation
![Page 3: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/3.jpg)
100x (+) Performance Difference – 2003 Custom C versus Relational Database
Confidential © 2012 Actian Corporation 3
26.2 28.1
0.2 0.6 0
5
10
15
20
25
30
MySQL DBMS 'X' C program Vectorwise
TPC-H 1 GB query 1 (runtime in s)
MySQL DBMS 'X' C program Vectorwise
![Page 4: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/4.jpg)
Some Numbers
Confidential © 2012 Actian Corporation 4
Traditional RDBMS: <200 MB/s per core Even these use MPP to I/O challenges
Vectorwise (lab environment): >1.5 GB/s per core Maximum throughput requirement is extremely high
Realistically (cost-effectively) only RAM can serve data quick enough
![Page 5: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/5.jpg)
What Hardware to Use
Confidential © 2012 Actian Corporation 5
CPU
Memory
Storage I/O and capacity
Requirements Budget
![Page 6: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/6.jpg)
Hardware Considerations – MEMORY
Confidential © 2012 Actian Corporation 6
Ideally frequently-accessed data should fit in memory May be all data
May be a small portion of the data
Note: data is compressed in memory buffer
• 3x – 5x compression ratios are common
Query execution should all take place in memory Operations against larger data sets require more memory
Consider query concurrency
“Spill to disk” is supported but should be a last resort
![Page 7: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/7.jpg)
Hardware Recommendation
Confidential © 2012 Actian Corporation 7
CPUs Use CPUs with higher clock rate for better raw throughput
Use more cores for higher throughput
Higher power CPUs are faster
Memory At least 8 GB per core (more is always better)
Storage Use as many drives as possible
Ensure sufficient capacity
Use the fastest drives available
• SAS over SATA, ideally 15k RPM
• SSDs are often not cost-effective relative to more memory
![Page 8: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/8.jpg)
Examples
Confidential © 2012 Actian Corporation 8
Small configuration (1 TB) Dell R620 Lenovo RD430
Medium configuration (single digit TBs) Dell R720 HP DL380 IBM x3650 Lenovo RD630
High-end configuration Dell R910 HP DL580 or DL980 IBM x3750
![Page 9: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/9.jpg)
Operating System Considerations
Confidential © 2012 Actian Corporation 9
Redhat SuSE
Ubuntu
Windows 7 (or higher) Windows 2008 (or higher)
64-bit
xfs, ext3, ext4
![Page 10: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/10.jpg)
Database Configuration
Confidential © 2012 Actian Corporation 10
Installation defaults are generally good May want to adjust column buffer size (default 25% of RAM)
May want to adjust processing memory (default 50% of RAM)
![Page 11: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/11.jpg)
Database Design
Confidential © 2012 Actian Corporation 11
Schema – no particular preference Single demormalized table, star schema, snowflake schema, 3rd normal form
Constraints Only on empty tables today… (to be addressed in Vectorwise 3.0)
Consider data loading order and impact
Indexes Note: clustered index-only today (“index-organized table”)
One per table
Consider incremental load
![Page 12: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/12.jpg)
Data Loading
Confidential © 2012 Actian Corporation 12
Initial load File-based bulk load through vwload or copy
Conversion into UTF8
Use tools Pentaho
Informatica
Talend
HVR
Attunity
![Page 13: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/13.jpg)
Data Loading
Confidential © 2012 Actian Corporation 13
Incremental load INSERT, UPDATE and/or DELETE
Append if possible
Batch if possible
Use COMBINE
Positional Delta Trees Memory considerations
Propagation to disk
Use tools
![Page 14: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/14.jpg)
Moving Window of Data
Confidential © 2012 Actian Corporation 14
Considerations COMBINE on a large table can be expensive
Mostly relevant for updates and deletes
Alternative: manual partitioning One table per period
Single view across all tables
![Page 15: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/15.jpg)
High Availability
Confidential © 2012 Actian Corporation 15
Hardware and OS best practices UPS, RAID
Vectorwise backup Only read-only, full backup
Consider periodic full backup and file incremental loads
Disaster recovery Dual load
Active/active possibility
![Page 16: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/16.jpg)
Monitoring
Confidential © 2012 Actian Corporation 16
OS monitoring CPU, memory utilization, I/O statistics
vwinfo data
Actian Director
DBA tools
![Page 17: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/17.jpg)
Agenda
17
Hardware
Operating system
Database configuration
Database design
Data loading
High availability
Monitoring
Confidential © 2012 Actian Corporation
More information in the Vectorwise Developer Guide: http://www.actian.com/images/white_papers/vw_developers_v2.5.pdf
![Page 18: A27 Vectorwise Performance Considerations_implementation_best_practices](https://reader033.vdocuments.site/reader033/viewer/2022052301/557567a9d8b42a2e248b49da/html5/thumbnails/18.jpg)
Confidential © 2012 Actian Corporation