in-memory technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 ·...
TRANSCRIPT
![Page 1: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/1.jpg)
1
In-Memory Technologie
Adastra
20.4.2016
Josef Pinkr, Consultant
cz.linkedin.com/in/josefpinkr
![Page 2: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/2.jpg)
In-memory technologies
3
Excel + Power BI add-insQuery, Pivot, View, Map
SharePointPower Pivot Gallery, Power View
ExcelData Mining
Power BI Desktop Power BI Portal
Azure ML
Power BI Mobile App
Analytics Platform System(APS)
Faster transactions, Faster Queries, Faster Insights
![Page 3: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/3.jpg)
In-Memory Technologies
4
Up to 30x faster transactions
In-memory objects
Row-store data format
Non/Durable data
Max. table size 2 TB
Many other improvements
Up to 10x query performance
Up to 10x data compression
Column-based data storage
• xVelocity (analytics engine)
Nonclustered columnstore index
• Updatable
• Filtered
Many other improvements
Billions of rows per second
with PowerPivot in Excel
Greater performance with
In-Memory Analysis Services
F A S T E R
![Page 4: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/4.jpg)
Business scenarios
5
• ODS / OLTP systems
• No reporting / No analysis
• Operational reporting
• OLTP reporting
• Static reporting
• Management reporting
• Data analysis
• BI / Ad-hoc reporting
![Page 5: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/5.jpg)
6
In-Memory OLTP
![Page 6: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/6.jpg)
In-Memory OLTP
7
• Up to 2 TB (from 256 GB)
• Support for LOB types
• Fully transactional
• ALTER TABLE
• Non/Durable
• Accessed Using T-SQL
• 1 – 8 indexes
• Columnstore index
• Foreign Key
• Check/Unique Constraints
• All collations
• Operate against memory-optimized tables
• Is compiled when it is created
• Support for LOB types
• OUTPUT clauses
• UNION / UNION ALL
• SELECT DISTINCT
• OUTER JOIN
• Subqueries in SELECT(EXISTS, IN)
• Must be schema bound
• Natively compiled storedprocedures
• Natively compiled scalaruser-defined functions
• Traditional T-SQL modules
Tables Stored procedures User-defined functions Triggers
• AFTER trigger
• On memory-optimized tables
![Page 7: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/7.jpg)
Performance Improvement of In-memory OLTP
Application Layer
SQL Server
OLTP system
Presentation Layer
INSERT SELECT
Application Layer
SQL Server
OLTP system
Presentation Layer
INSERT SELECT
Application Layer
SQL Server
OLTP system
Presentation Layer
INSERT SELECT
Memory-optimized tableMemory-optimized table
Natively compiled SP
2x 11x
Test: 4 Workloads \ 10 Threads \ 1000 Iterations
![Page 8: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/8.jpg)
10
Columnstore
![Page 9: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/9.jpg)
Columnstore
11
• Technology for storing, retrieving and managing data by using a columnar data format
• Better performance for new BATCH execution mode
• Preferred for data warehousing and analytics workloads
• Not the only index
• Physical storage for the entire table
• Updatable
• Filtered
• Compressed (from 2014)
• Secondary index on row-store table
• Contains a copy of part/all of the rows and columns
![Page 10: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/10.jpg)
Nonclustered Columnstore Index (NCI)
12
Key Points
• Create an updateable non-clustered columnstore index (NCCI) for analytics queries
• Drop all other indexes that were created for analytics.
• No OLTP Application changes.
• ColumnStore index automatically keeps up with DML operations
• Query Optimizer will choose columnstore index where needed
Relational Table (disk-based)(Clustered Index/Heap)
Btree IndexD
ele
te b
itm
ap
Dele
te B
uff
er
Delta rowgroups
![Page 11: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/11.jpg)
Minimizing Columnstore Index overhead – Filtered index
13
Key Points
• Create Columnstore only on cold data – using filtered predicate to minimize maintenance
• Analytics query accesses both columnstore and ‘hot’ data transparently
• Example:
– Order Management Application: create nonclustered columnstore index … where order_status = ‘SHIPPED’
Relational Table (disk-based)(Clustered Index/Heap)
Btree IndexD
ele
te b
itm
ap
Dele
te B
uff
er
HOT
Delta rowgroups
![Page 12: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/12.jpg)
Minimizing Columnstore Index overhead – Compression delay
14
Syntax – delay in minutes:
Create nonclustered columnstore index <name>
on <table> (<columns>)
with (compression_delay = 30)
Relational Table (disk-based)(Clustered Index/Heap)
Btree Index
HOT
Delta rowgroups
Dele
te b
itm
ap
Dele
te B
uff
er
Key Points
• Delta RG is only compressed after ‘Compression_Delay’ duration
• Minimizes/Eliminates index fragmentation
![Page 13: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/13.jpg)
16
In-Memory Analytics
![Page 14: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/14.jpg)
Operational analytics
17
Benefits
• No Data Latency
• No ETL
• No Separate DW
Challenges
• Minimizing Impact on OLTP
workload
• Delivering Performant Analytics
It is NOT for
• Extreme analytics – needs star-
schema, pre-aggregated cubes
Presentation Layer
Application Layer
SQL Server
OLTP system
Add columnstore index
(Create in-memory table)
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
Select ProduceName, ExpiryDate, Quantity, sum (sales)
From <transactions>
Where ProduceType =‘perishables’
Group By ProduceName, ExpiryDate
Reporting and analytics
Operational analytics
ETL
SQL Server Analysis Services
SSAS Tabular
Data Mart
![Page 15: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/15.jpg)
SSAS
Data refresh
Reporting and analytics
SQL Server
Scenario 2
Data Stage
Analytical query
Data upload (L1)
In-MemoryAnalytics in Data Warehouse
18
Tabular Mode
Reporting and analytics
DWH / SQL Server
Scenario 1
Disk based tables
Data Stage
Data upload (L1)
Analytical query
Data
Late
ncy
Response T
ime
Data
Late
ncy
Response T
ime
Columnstore indexes
(non/clustered)
Reporting and analytics
SQL Server
Scenario 3
Data Stage
Analytical query
Data upload (L1)
Columnstore indexes
(non/clustered)
25x
+ 3%
10M rows \ Aggregations: Sum, Max, DistinctCount
40x
+ 50%
![Page 16: In-Memory Technologiedownload.microsoft.com/documents/cs-cz/enterprise/... · 4/20/2016 · In-MemoryTechnologies 4 Up to 30x faster transactions In-memory objects Row-store data](https://reader034.vdocuments.site/reader034/viewer/2022043006/5f91565f19e85f4dc427efe2/html5/thumbnails/16.jpg)
ADASTRA CZECH REPUBLICAdastra, s.r.o.
Karolinská 654/2, 186 00 Praha 8
Tel.: +420 271 733 303
www.adastra.cz
ADASTRA GROUP North America8500 Leslie St.
Markham, Ontario, L3T 7M8
Tel: +1 905 881 7946
Restrictions for public release and use:This document can comprise confidential information. As such it may not, without Adastra’s prior consent, be copied or transferred.
Important:All brands and names of products given in this documentation are or can be registered trademarks of their owners.© 2016 Adastra, all rights reserved.
19
Thank you!