keeping it tidy - how to warehouse your sharepoint data
DESCRIPTION
We all know that SharePoint is an excellent tool for gathering information and collaborating. What it is less good at it maintaining large quantities of that information over long periods of time. The good news is that SQL Server is very good at exactly that. In the spirit of choosing the right tool for the right job, this session will present a methodology for moving your historical SharePoint information from SharePoint and into a SQL Server based data warehouse. From there, we will cover several method for then surfacing data back in SharePoint in context.TRANSCRIPT
John WhiteChief Technology OfficerUnlimitedViz Inchttp://whitepages.unlimitedviz.com@diverdown1964
Keeping It TidyHow to Warehouse Your SharePoint Data
John WhiteCTO/Co-Founder of UnlimitedVizSharePoint Server MVP, SQL Server v-TS
[email protected]://whitepages.unlimitedviz.com@diverdown1964
3© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Agenda
SharePoint Data Storage Data Warehousing Moving SharePoint Data Surfacing Warehoused Data in SharePoint Q&A
SharePoint Data Storage
The Problem
5© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Excel Data Storage
List based data Powerful Features Massive adoption Good usability
NOT a database Not relational Capacity limits Not authoritative
Excellent BI Client Surface the data.. DON’T store it
SharePoint Content Databases
7© DEVintersection. All rights reserved.
http://www.DEVintersection.com
SharePoint Content Databases
~160 fields
8© DEVintersection. All rights reserved.
http://www.DEVintersection.com
SharePoint Capacity Boundaries
Boundaries and Limits - http://technet.microsoft.com/en-us/library/cc262787 200 GB per content DB (general usage scenarios - or up to 4 TB) List view threshold – 5,000 simultaneous items List view lookup threshold – 8 join operations per query Items per content database – 60 million
Demo – WorkflowHistory List
Topic
Data Warehousing
ONE DOES NOT SIMPLY
RETURN HIS RAW DATA FROM THE DATABASE
12© DEVintersection. All rights reserved.
http://www.DEVintersection.com
BI Architecture 101
Data Marts
Extract, Transform, and Load (ETL)
Middleware Server(s)
DataWarehouse
Storage Design and Visualization
Data Cubes and Tabular
Models
E
T
L
Reporting Server(s)
BI and Designer Clients
Source data
13© DEVintersection. All rights reserved.
http://www.DEVintersection.com
EE
Microsoft enterprise (classic) BI
SQL Server DBSQL Server Integration Services (SSIS)
SharePoint (with)• Excel Services• PowerPivot for
SharePoint• SSRS SharePoint Mode• PerformancePoint
SQL Server DB
Storage Design and Visualization
SQL Server Analysis Services
Multidimensional and Tabular modes
L
SQL Server Reporting Services (SSRS)
ExcelSQL Data ToolsReport Builder3rd party tools
ETL
E
T
Source data
14© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Team BI and SharePoint Dashboards
Power Pivot Worksheets• Pivot Tables and
Charts• Power View
Data Marts and other
Data Cubes and Tabular
Models
Standard Worksheets• Pivot Tables and
Charts
PerformancePoint Reports• Analytic Charts and Grids• Decomposition trees
SQL Server Reporting Services Reports• Standard• Power View
PerformancePoint Scorecards and KPIs
15© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Why Warehouse?
Optimize for End Goal Multidimensional Flattened
Usability Reduced Load on Source Systems Security Requirements Licensing Requirements SharePoint (Performance)
Topic
Moving SharePoint Data
17© DEVintersection. All rights reserved.
http://www.DEVintersection.com
ETL Tools
SQL Server Integration Services Comes with ALL Editions of SQL Server (except Express)
SQL Server Data Tools http://www.microsoft.com/en-ca/download/details.aspx?id=36843 (For VS2012)
SharePoint List Source and Destination (Codeplex Project) http://sqlsrvintegrationsrv.codeplex.com/releases/view/17652 (Download) http://
dataqueen.unlimitedviz.com/2011/06/how-to-use-a-sharepoint-list-as-a-data-source-in-your-ssis-package/ (How to)
OData Source for Microsoft SQL Server 2012 (Official) http://www.microsoft.com/en-us/download/details.aspx?id=42280 (Download) http://whitepages.unlimitedviz.com/2014/03/using-the-odata-source-connector-with-sharepoint-o
nline-authentication/ (How to)
http://www.microsoft.com/en-pk/download/details.aspx?id=35585 – Required for Office 365 authentication
Moving SharePoint Data
Demo
Topic
Accessing Warehoused Data
20© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Data Consumption
Business Connectivity Services External List All SharePoint Editions (yes..Foundation)
SQL Server Reporting Services Advanced Reporting All Editions of SQL Server (even Express!) ..except Express doesn’t do SP Integrated Mode
Excel/Power Pivot Excel Services requires Enterprise SharePoint PowerPivot for SharePoint requires SQL Server Standard or better
Accessing Warehoused Data
Demo
22© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Considerations
Transformations Mashups Scheduling Data latency
In Conclusion
24© DEVintersection. All rights reserved.
http://www.DEVintersection.com
Q&A
@diverdown1964whitepages.unlimitedviz.com