Moving a 100TB Application from Oracle to Microsoft SQL Server
Praveen SrivatsaDirector, Asthrasoft ConsultingMicrosoft Regional Director and MVP
DBI311
Agenda
NMS (Large Load) Case Study Comparison between Oracle and SQL implementationScaling Points and Optimizations
Async operations with Service BrokerDistributing data with Table partitionsData CompressionOther optimization options
Case StudyExisting NMS application
The NMS application collects, logs and analyzes the logs of various digital equipment in the networkUsed for mobile towers, wi-max networks, wifi-networks and data centersIt’s agents collects logs 24x7 from all equipment and pumps the same to a central store.The logs are analyzed for security breaches or attacks as well as to monitor the health of the various equipment.
Existing NMS application
LogManager
1. Collect the data
2. Analyze the data
3. Log and Report
Agent
Agent
Existing NMS application
LogManager
Agent
Agent
0.5 KB/record
2500 records/ agent
10 agents
= 25K r/s(12. 5 MB/s)
25K r/s == 12.5 MB/s
1.5M r/min == 750 MB/min
90M r/hr == 45 GB/hr
2.16B r/day == 1.08 TB/day
Reporting on 100 days of data
Case StudyCurrent architecture with Oracle
RAW TABLES
Reporting Dashboard1. Agents collect data
and write to CSV
2. Oracle BULK Loads data
3. Data is indexed and analyzed
4. Materialized Views are created for reporting
Case StudyServer Configurations
App Server2 Quad Core (8 Cores)32 GB RAM1TB HDDRedhat LINUXPython based App
DB Server2 Quad Core (8 Cores)32 GB RAM1 TB HDDSolaris 10Oracle R2 10G
200 TB External storage on RAID 5 (120 TB usable)
Case StudyKey Challenges
Bulk Data Import is very fast, but the data is not ready for analysis or reporting till indexes/materialized views are builtWith the data inserts going on, the indexing and analysis is really slow – so differed to end of the day when a new table is created (EOD : 13 hrs)Data querying for reporting now has to be done across multiple tables (upto 100 tables with a UNION) adding a lot of overheadLAG from data collection to reporting was anywhere from 13 hrs to 37 hrs
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Compressed Tables
Partitioned Tables
WCF
Reporting Dashboard
Case StudyMoving to SQL Server
Collection with data co-relation33 records/sec(2.85 M/day)
WCF
Reporting Dashboard
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Collection with differed co-relationBatch of 100 recs25 batches/sec/Q (2500/s)X 10 Queues (25000/s)
2.16 B/dayWCF
Reporting Dashboard
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Partitioned Tables
Parallelized data co-relation5 batches (100r)/sec = 500r/sX 5 Readers = 2500 r/sec/QX 10 Q = 25000 r/s
2.16 B/day
WCF
Reporting Dashboard
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Compressed Tables
Partitioned TablesRaw data saving due
to compression28%
WCF
Reporting Dashboard
Case StudyServer Configurations
App Server2 Quad Core (8 Cores)32 GB RAM500GB HDDWindows Server 200R2WCF Based CollectorPython App for reporting
DB Server2 Quad Core (8 Cores)64 GB RAM500GB HDDWindows Server 2008R2SQL Server 2008R2
200 TB External storage on RAID 5 (120 TB usable) on Windows Storage Server
Case StudyKey Benefits with SQL Server
Data transfer was continuous and could be parallelized using more queuesQuerying and analysis did not slow down the data collectionMore complex analysis and reporting could be added onto the logsLag between collection and reporting came down to 1hr 12 mins
Case StudyAdditional Benefits with SQL Server
Historical data was compressed, reporting data was uncompressed, but partitionedDB Snapshots every hour provided a shorter window of analysis and reportingCLR Procedure provided advanced rules operations on the data packets
demo
NMS Walkthrough
End to End walkthroughof the simulated app
Scale PointsService Broker Advantages
Is a queued operations like MSMQ or MQSeriesScales for rapid insertsCan create multiple parallel queuesCan use activation to process the queueGives a guaranteed onetime only deliveryCan transmit data across DB instances
OptimizingService Broker
Configure multiple packets per conversationBulk load multiple records into a single messageTransform and bulk insert from SSB XML to tableOptimize packet size of SSB Queue
demo
Service Broker Demo
Optimizing SSB performance
Scale PointsTable Partitions and Compressions
PartitionSeparates logical units of data without changing codeAllows fast archiving with the SWITCH Partition operationAllows partition level management rather than at table
CompressionCompressed 20-35% dataLoading data, backups, traversal over network all benefittedPage-Row level configurations possible
OptimizingTable Partitions and Compressions
PartitionCreate a logical partition switch to parallelize insertsAlign partitions to number of disks and procsAlign indexes to the table partitions
CompressionEnable compression at a page levelPlan table row size in alignment with the page size
demo
Table Partition Demo
Inserting into multiple partitions in parallel
Scale PointsOther tweaks
DB SnapshotsGives a point-in-time read-only view of the data
XML operationsFast and easy XML Bulk operations on SSB Queue data
CLR ProceduresCompiled procedures for co-relation and alerts
Summary
Key benefits of moving to SQL Server from the current Oracle architecture included
A shorter lag for critical reporting (from 13-37hrs to about 75 min)
A scale-out model allowing for even greater scalability
An integrated set of technologies (relational database, queuing technologies, compression options, reporting solutions) to work with
Takeaways
SQL Server leverages a scale out modelIt does this by using parallel operations at multiple levels
SQL Service Broker provides a robust queuing mechanismAsync queues provides a horizontal scale operations even across multiple databases
Table Partitions allows tables to scale to billions of recordsEffective segregation allows single tables to handle large volumes of data effectively
Leveraging SQL Server featuresDB Snapshots provides a point in view of the dataCompression allows us optimize our data storage
Resources
Architecting Service Broker Applicationshttp://technet.microsoft.com/en-us/library/aa964144.aspx
Partitioning Tables and Indexeshttp://msdn.microsoft.com/en-us/library/ms345146
My website, blog and EMailweb : http://www.asthrasoft.comblog : http://studidesk.asthrasoft.com/srivatsapraveenemail : [email protected]
Related Content
DBI307 | Automating database migration to Microsoft SQL Server
DBI315 | Microsoft SQL Server in virtualization and private cloud
DBI330 | Can your BI Solutions scale?
Database Platform (DAT) Resources
Try the new SQL Server Mission Critical BareMetal Hand’s on-Labs
Visit the updated website for SQL Server® Code Name “Denali” on www.microsoft.com/sqlserver and sign to be notified when the next CTP is availableFollow the @SQLServer Twitter account to watch for updates
Visit the SQL Server Product Demo Stations in the DBI Track section of the Expo/TLC Hall. Bring your questions, ideas and conversations!
• Microsoft® SQL Server® Security & Management • Microsoft® SQL Server® Optimization and Scalability• Microsoft® SQL Server® Programmability • Microsoft® SQL Server® Data Warehousing• Microsoft® SQL Server® Mission Critical • Microsoft® SQL Server® Data Integration
Resources
www.microsoft.com/teched
Sessions On-Demand & Community Microsoft Certification & Training Resources
Resources for IT Professionals Resources for Developers
www.microsoft.com/learning
http://microsoft.com/technet http://microsoft.com/msdn
Learning
http://northamerica.msteched.com
Connect. Share. Discuss.
Complete an evaluation on CommNet and enter to win!
Scan the Tag to evaluate this session now on myTech•Ed Mobile