sql server workshop for developers - visual studio live! ny 2012
DESCRIPTION
TRANSCRIPT
SQL Server Workshopfor Developers
SQL Server Workshopfor Developers
Leonard LobelCTO, Sleek Technologies
Principal Consultant, Tallan
Andrew BrustCEO and FounderBlue Badge Insights
Level: Intermediate
May 14, 2012
• CEO and Founder, Blue Badge Insights• CTO, Tallan, Inc.• Member, Microsoft BI Partner Advisory Council• Microsoft Regional Director, MVP• Co-chair VSLive! and over 15 years as a speaker• Founder, Microsoft BI User Group of NYC
– http://www.msbinyc.com
• Co-moderator, NYC .NET Developers Group– http://www.nycdotnetdev.com
• “Redmond Review” columnist for Visual Studio Magazine and Redmond Developer News
• brustblog.com, Twitter: @andrewbrust
Meet Andrew
Meet LenniLeonard Lobel• CTO & Co-Founder
– Sleek Technologies, Inc.
• Principal Consultant– Tallan, Inc.
• Microsoft MVP– SQL Server
• .NET consultant and trainer• Speaker• Author• Programming since 1979
Contact• Email: [email protected] • Blog: lennilobel.wordpress.com• Twitter: @lennilobel
sleek technologies
Read All About It!
• Part I– Overview
• Part II– T-SQL Enhancements
• Part III– Business Intelligence
• Part IV– Beyond Relational
• Demos, demos, demos!–http://sdrv.ms/VSLiveNY2012SQL
Agenda
http://sdrv.ms/VSLiveNY2012SQL
(case sensitive!)
Download Slides and Code
OVERVIEW
PART I
• SQL Server 2012 (Version 11)– Newest version of SQL Server– Released March 7, 2012
• SQL Server 2008 R2 (Version 10.5)– Adds powerful BI features to SQL Server 2008– Released April 21, 2010
• SQL Server 2008 (Version 10)– Released August 6, 2008
• SQL Server 2005– Revolutionary upgrade from SQL Server 2000– Integration of multiple services (SSRS, SSAS, SSIS) with the RDBMS
• Before SQL Server 2005…– The relational database engine was the product– Added value features (Reporting, OLAP, DTS) through a patchwork of
optional add-ons
SQL Server Versions
• SQL Server 2012 (Version 11)– Newest version of SQL Server– Released March 7, 2012
• SQL Server 2008 R2 (Version 10.5)– Adds powerful BI features to SQL Server 2008– Released April 21, 2010
• SQL Server 2008 (Version 10)– Released August 6, 2008
• SQL Server 2005– Revolutionary upgrade from SQL Server 2000– Integration of multiple services (SSRS, SSAS, SSIS) with the RDBMS
• Before SQL Server 2005…– The relational database engine was the product– Added value features (Reporting, OLAP, DTS) through a patchwork of
optional add-ons
SQL Server Versions
• The latest major release of SQL Server• Mission Critical Platform
– HADR (High-Availability Disaster Recovery)
• IT and Developer Productivity– SQL Server Data Tools (SSDT)– T-SQL enhancements– Beyond Relational enhancements (FileTable, FTS
improvements)– Geospatial improvements (circular data, full globe, performance)
• Pervasive Insight– Columnstore Indexing (xVelocity)– BI Semantic Model (PowerPivot technology) comes to Analysis
Services– Real ad hoc reporting and data visualization in Power View– Data lineage and data quality (DQS)
Introducing SQL Server 2012
• Stack Review– The MS BI Stack– The SQL Server 2008 R2 “sub-stack”– New SQL Server 2012 Components
• Analysis Services and OLAP– Dimensional Concepts– Analysis Services cube design– Overview of ADO MD.NET and other APIs
BI Foundation
• Presentation Layer– Excel BI– PowerPivot and Excel Services
Including new PowerPivot features in SQL Server 2012 – Analysis Services tabular databases (SQL Server 2012)– Power View (SQL Server 2012)– Reporting Services and Report Builder– PerformancePoint Services (brief)
• Overview: Other New R2/2012 Components– Master Data Services, Data Quality Services– StreamInsight
BI Delivery
• SSDT – Code-named “Juneau”– Next generation IDE for SQL Server development– Declarative, model-based development in connected, offline, and
cloud scenarios– New project type: SQL Server Database Project– Visual Studio shell, solutions, projects + (partial) SSMS/Azure
functionality + full BIDS
• Not intended to replace SSMS– SSMS still the primary dba tool for managing SQL Server
• Intended to replace VS DbPro (aka “data dude”)– But not ready to, as it still lacks total feature parity– Missing data generation, data compare, unit testing
• Separate Web Platform Installer Download– Updates to ship out-of-band with VS and SQL Server releases
Introducing SQL Server Data Tools
Database Model
Declarative Model-Based Development
SQL Server 2005, 2008, 2008 R2,
2012
Database Snapshot File (.dacpac)
SQL Server Data Tools (SSDT)
On-Premise DataCenter
Offline Dev/Test Cloud
Version History
SSDT Projects
Relational Engine
Analysis Services
Reporting Services
Integration Services
SQL Server Object Explorer
Static Analysis and Validation
Database Publish
T-SQL Language Services
Power Buffer Editing
Table Designer Schema Compare
Local Database Runtime
T-SQL Debugging SQL CLR
SSDT
T-SQL ENHANCEMENTS
PART II
TABLE-VALUED PARAMETERS
• Process a set of rows as a single entity– Similar to temp tables, table variables and CTEs– Example: INSERT an entire order (header & details) with only two
parameters and one stored procedure call
• Populate a TVP, and then pass it around– It’s a single parameter– Pass from procedure to procedure on the server– Pass from client to server across the network
• Based on User-Defined Table Types– Defines the schema, just like an ordinary table– Simply declare a variable as the type to get a TVP
• Stored in tempdb– Created and destroyed automatically behind the scenes– Can be indexed
Table-Valued Parameters
CREATE TYPE CustomerUdt AS TABLE (Id int, CustomerName nvarchar(50), PostalCode nvarchar(50))
DECLARE @BestCustomers AS CustomerUdt
Creating a User-Defined Table Type
Table-Valued Parameters
• TVPs are read-only, once populated and passed– You must apply the READONLY keyword when declaring
TVPs in stored procedure signatures– OUTPUT keyword cannot be used– You cannot update, insert or delete
• No ALTER TABLE…AS TYPE statement– To change the schema, it must be dropped and re-created– All dependent objects must also be dropped and re-created
• Statistics are not maintained on TVPs
TVP Limitations
MERGE
• Four statements in one– SELECT– INSERT– UPDATE– DELETE
• And even more…– OUTPUT clause– INSERT OVER DML
• Operates on a join– Between source and target– Type of join based on merge clause
• Start using it now– 100% compatible with existing business logic– Existing triggers continue to work
MERGE
MERGE target USING source ON join WHEN MATCHED UPDATE | DELETE WHEN NOT MATCHED [BY TARGET] INSERT WHEN NOT MATCHED BY SOURCE UPDATE | DELETE;
MERGE Syntax
MERGE
INSERT OVER DML
• INSERT, UPDATE, DELETE, and MERGE all support the OUTPUT clause– Captures before-and-after snapshots of modified data via
INSERTED and DELETED pseudo-tables (just like triggers)– MERGE adds $action virtual column (returning INSERT,
UPDATE or DELETE)
• OUTPUT INTO can capture the change data to a table or table variable– Suffers from one limitation – no filtering– Solution – use INSERT OVER DML
DML Output
INSERT INTO target(columns) SELECT columns FROM (DML statement with OUTPUT) CHANGES(columns) WHERE filter
INSERT OVER DML Syntax
DATE AND TIME TYPES
• Start using these four data types– date– time– datetime2– datetimeoffset
• Stop using these two data types– datetime– smalldatetime
• Enhanced storage, portability and functionality– Greater range and precision– More efficient compacted storage– Time zone awareness– All traditional functions support the new types– ODBC, OLE-DB, ADO.NET, SSIS, SSAS, SSRS, Replication
Improved Date and Time Support
• Only use what you need– Eliminate extraneous storage when only date or time is needed– Better performance for date-only manipulations and
calculations
• For example,DECLARE @DOB AS dateDECLARE @MedsAt AS time(0)
Separate Date and Time Types
• Value ranges align with .NET and Windows• Date Values
– From 1/1/0001 (DateTime.MinValue)– Through 12/31/9999 (DateTime.MaxValue)– Legacy datetime type limited to 1/1/1753 through 12/31/9999
• Time Values– Up to 100ns (10-millionth of a second) precision– datetime accurate only to roughly hundredth of millisecond,
smalldatetime to the minute
More Portable Dates and Times
• datetimeoffset type– Same range and precision as datetime2
• Includes the time zone– Stores an offset ranging from -14:00 to +14:00– Does not support DST (daylight savings time)
• Store local date/time in different regions of the world– Values appear to go in and out as local dates/times
• Internally stored in UTC– Automatically converted and treated as UTC for comparisons,
sorting and indexing
• You append the time zone…– …and SQL Server handles the conversions to and from UTC under
the covers automatically
Time Zone Awareness
• Date values compacted into 3 bytes– One byte less than 4-byte date portion of datetime– Greater range in less space!
• Time values consume 5 bytes at most– Supports 100-ns accuracy
• Pay less for less– Reduced storage for times that don’t require high accuracy– Specify a scale parameter (0-7) on time, datetime2 or datetimeoffset
types– 0 = No decimal precision in 3 bytes– 7 = Greatest decimal precision (100-ns) in 5 bytes– Example: DECLARE @FeedingTime time(0)– Differing scales compatible for comparison
Date/Time Accuracy and Storage
Date and Time Types
• Windowing (OVER Clause) Enhancements• New T-SQL Functions in SQL Server 2012• The THROW Statement• Server-Side Paging• The SEQUENCE Object• Metadata Discovery• Contained Databases
T-SQL Enhancements(SQL Server 2012)
Introducing OVER
• OVER Clause– Exposes a window over the result set for each row– Added in SQL Server 2005, along with the ranking
functions ROW_NUMBER, RANK, DENSE_RANK, NTILE
• Can also be used with aggregate functions– SUM, COUNT, MIN, MAX, AVG– Doesn’t require GROUP BY
OVER
AmountAcctId TxnDate
1 3/10/2012
1 3/22/2012
500
250
1 3/24/2012
1 3/26/2012
75
125
2 3/11/2012
2 3/15/2012
500
50
2 3/22/2012
2 3/24/2012
5000
550
2 3/27/2012
3 3/15/2012
95
600
3 3/22/2012
3 3/23/2012
25
125
AmountAcctId TxnDate
1 3/10/2012
1 3/22/2012
500
250
1 3/24/2012
1 3/26/2012
75
125
2 3/11/2012
2 3/15/2012
500
50
2 3/22/2012
2 3/24/2012
5000
550
2 3/27/2012
3 3/15/2012
95
600
3 3/22/2012
3 3/23/2012
25
125
OVER with PARTITION BY
AmountAcctId TxnDate
1 3/10/2012
1 3/22/2012
500
250
1 3/24/2012
1 3/26/2012
75
125
2 3/11/2012
2 3/15/2012
500
50
2 3/22/2012
2 3/24/2012
5000
550
2 3/27/2012
3 3/15/2012
95
600
3 3/22/2012
3 3/23/2012
25
125
AmountAcctId TxnDate
1 3/10/2012
1 3/22/2012
500
250
1 3/24/2012
1 3/26/2012
75
125
2 3/11/2012
2 3/15/2012
500
50
2 3/22/2012
2 3/24/2012
5000
550
2 3/27/2012
3 3/15/2012
95
600
3 3/22/2012
3 3/23/2012
25
125
OVER with PARTITION BYand ORDER BY
AmountAcctId TxnDate
1 3/10/2012
1 3/22/2012
500
250
1 3/24/2012
1 3/26/2012
75
125
2 3/11/2012
2 3/15/2012
500
50
2 3/22/2012
2 3/24/2012
5000
550
2 3/27/2012
3 3/15/2012
95
600
3 3/22/2012
3 3/23/2012
25
125
AmountAcctId TxnDate
1 3/24/2012
1 3/26/2012
75
125
1 3/22/2012
1 3/10/2012
250
500
2 3/11/2012
2 3/15/2012
50
95
2 3/22/2012
2 3/24/2012
500
550
2 3/27/2012
3 3/15/2012
5000
25
3 3/22/2012
3 3/23/2012
125
600
• OVER with PARTITION BY– Optionally groups the result set into multiple windows
• OVER with ORDER BY– Specifies the row sequence in each window– Required for the ranking functions– Not previously supported for the aggregate functions
• Now in SQL Server 2012– OVER with ORDER BY now supported with aggregate
functions– Window “framing” with ROWS and RANGE– Eight new analytic windowing functions
Window Partitioning and Ordering
Windowing(OVER Clause)
• FIRST_VALUE– Returns a column value from the first row of the window
• LAST_VALUE– Returns a column value from the last row of the window
• LAG– Returns a column value from a previous row of the window
• LEAD– Returns a column value from a subsequent row of the window
• PERCENT_RANK– Calculate percentile as (RANK – 1) / (N – 1)
• CUME_DIST– Calculate percentile as RANK / N
• PERCENTILE_DISC– Returns a discreet column value at the specified percentile
• PERCENTILE_CONT– Returns a value based on the scale of column values at the
specified percentile
New T-SQL Analytic Functions
• Date/Time– DATEFROMPARTS– TIMEFROMPARTS– DATETIME2FROMPARTS– DATETIMEOFFSETFROMPARTS– DATETIMEFROMPARTS– SMALLDATETIMEFROMPARTS– EOMONTH
• Math– LOG (changed)
More New T-SQL Functions
• Logical– CHOOSE– IIF
• String– CONCAT– FORMAT
• Conversion– TRY_CONVERT– PARSE– TRY_PARSE
New SQL Server 2012 T-SQL Functions
• SQL Server 2005 added TRY/CATCH – Borrowed from .NET’s try/catch model– Vast improvement over repeatedly testing @@ERROR– Still used RAISERROR for generating errors
• SQL Server 2012 adds THROW– Recommended alternative way to generate your own errors– Does not entirely replace RAISERROR
• Two usages for THROW– With error code, description, and state parameters (like
RAISERROR)– Inside a CATCH block with no parameters (re-throw)
The THROW Statement
THROW vs. RAISERROR
THROW RAISERRORCan only generate user exceptions (unless re-throwing in CATCH block)
Can generate user (>= 50000) and system (< 50000) exceptions
Supplies ad-hoc text; doesn’t utilize sys.messages
Requires user messages defined in sys.messages (except for code 50000)
Doesn’t support token substitutions Supports token substitutions
Always uses severity level 16 (unless re-throwing in a CATCH block)
Can set any severity level
Can re-throw original exception caught in the TRY block
Always generates a new exception; the original exception is lost to the client
Error messages are buffered, and don’t appear in real-time
Supports WITH NOWAIT to immediate flush buffered output on error
THROW
• New result paging keywords• Append to ORDER BY clause• Limits the number of rows returned
• OFFSET @start ROWS• The first result row to return (zero-based)
• FETCH NEXT @count ROWS– The number of rows to return
Server-Side Paging
Server-Side Paging
• Sequential Number Generator– As found in Oracle and DB2– Alternative to using IDENTITY for assigning new primary keys
• Advantages over IDENTITY– SET IDENTITY INSERT ON/OFF not needed for inserts– Can obtain next sequence value without performing an insert
• Create a Sequence Object
• Retrieve Sequence Values
Sequences
CREATE SEQUENCE MySequence START WITH 1 INCREMENT 1 MINVALUE 1 NO MAXVALUE
INSERT INTO MyTable(Id, ...) VALUES(NEXT VALUE FOR MySequence, ...)
Sequences
• New system stored procedures and data management views
• sys.sp_describe_first_result_set– Analyzes a T-SQL statement and returns information describing the
schema of the statement’s result set
• sys.dm_exec_describe_first_result_set– Similar, but implemented as a table-valued function to support filtering
• sys.dm_exec_describe_first_result_set_for_object– Similar, but accepts an OBJECT_ID of a T-SQL object in the database
to be analyzed, rather than a T-SQL statement
• sys.sp_describe_undeclared_parameters– Analyzes a T-SQL statement and returns information describing the
parameter(s) required by the statement
Metadata Discovery
Metadata Discovery
Contained Databases
• Databases are not entirely self-contained– They have external instance-level dependencies– Logins, collations, tempdb, linked servers, endpoints, etc.
• SQL Server 2012 provides partial containment– Makes databases more portable– Enables (but does not enforce) containment
• Can store logins (with passwords) in the database– Users authenticate directly at the database level
• sys.dm_db_contained_entities– Discovers threats to containment
Contained Databases
BUSINESS INTELLIGENCE
PART III
MICROSOFT’S BI STACK AND SQL SERVER’S ROLE IN IT
Microsoft Business Intelligence
Business User Experience
Data Infrastructure and BI PlatformAnalysis ServicesReporting ServicesIntegration ServicesMaster Data ServicesData MiningData Warehousing
Business Collaboration PlatformDashboards & ScorecardsExcel ServicesWeb based forms & workflowCollaborationSearchContent ManagementLOB data integration
Familiar User Experience Self-Service access & insightData exploration & analysisPredictive analysisData visualizationContextual visualization
Business Collaboration Platform
Information Platform
SQL Server 2008 BI Components
But Wait, There’s More!
• R2: PowerPivot• R2: Report Parts in SSRS• 2012: Analysis Services Tabular mode
– And corresponding improvements in PowerPivot
• 2012: Power View• 2012: Data Quality Services
• How to get through it all? Here’s the menu…
The Appetizer
• Learn Data Warehousing/BI terms and concepts.
The Main Course
• Build a multidimensional cube– Query in Excel
• Build a PowerPivot model– Query that from Excel too.– Publish to SharePoint
• Upsize the PowerPivot model to SSAS tabular model– Add advanced features– Query from Excel
• Analyze tabular model from Power View
Dessert
• Reporting Services– Report Builder and Report Parts
• PerformancePoint Services• Overviews of
– Data Quality Services– Master Data Services– StreamInsight
BI CONCEPTS, ANALYSIS SERVICES
Business Intelligence
Preparing For Business Intelligence
• Transactions• Process
Transaction Database
• Data• Relationships• Analysis
Data Warehouse
Dimensional Model
• Measure• Dimension• Hierarchy• Grain• Star Schema
Star Schemas
• Physical data model• Central fact table• Multiple dimension
tables– Used to constrain fact table
queries TotalSales
Country
Year
SalesPersonProduct
Shipper
Example Data Request
• Get Total Sales By State, By Month for a Calendar Year For Country = USA and Calendar Year = 1996
Data Warehouse Query
Data Migration
• Transactions• Process
Transaction Database
• Data• Relationships• Analysis
Data
Warehouse • Multi-dimensional
• Hierarchical
OLAP Database
SQL Server Analysis Services
• Built for analysis• It is free with SQL Server• And you can use the Microsoft stack that you
know and love
From Data Warehouse to OLAP
• Measure• Dimension
– Can have Hierarchies
• Cube
Fact Table
Measures
Dimensions
Aggregations
Building OLAP Cube With BIDS
• Business Intelligence Development Studio– AKA Visual Studio
• Business Intelligence Projects– Analysis Services Project Type
Add Data SourceAdd Data Source ViewAdd Cube
Add Dimensions
Add MeasuresDeploy the Cube
Creating anSSAS ProjectThe Basics
Advanced OLAP
• Calculated Members• Key Performance Indicators• Perspectives• And:
– MDX– Actions– Partitions– Aggregations– Translations
QUERYING CUBES FROM YOUR APPLICATIONS
SSAS Interfaces
AMOADOMD.NET
Analysis Server (msmdsrv.exe)
OLAP Data Mining
Server ADOMD.NET
.NET Stored Procedures Microsoft Algorithms Third Party Algorithms
XMLAOver TCP/IP
OLEDB for OLAP/DM ADO/DSO
XMLAOver HTTP
Any Platform, Any Device
C++ App VB App .NET App Any App
WAN
DM Interfaces
ADOMD.NET Client
PRESENTATION TECHNOLOGIES
Presenting Your CubePerformancePoint Services
Cube
Excel Services
Excel
SQL Server
Tera- Data
Oracle DB2
Reporting Services
The SSAS/Excel/SharePoint Loop
Build models with SSAS Multidim’l, Tabular or
PowerPivot
Query from Excel
PivotTables and Charts
Publish to SharePoint(via Excel
Services) and query in the
browser
Visualize + Analyze with
SSRS/Excel Services/
PerformancePoint Services
OR Power View
EXCEL BI
PivotStuff
• PivotTable, and linked charts (sometimes referred to as PivotCharts) work extremely well with OLAP cubes
• How to create a PivotTable:– Ribbon’s Data tab (From Other Sources button/From Analysis
Services option or Existing Connections button)– Insert tab (PivotTable “split button”)
• How to insert a chart– PivotChart button on PivotTable Tools/Options tab– Several others
PivotCharts and PivotTables
Formula Language CUBE Functions
• CUBEMEMBER and CUBEVALUE– Also CUBEKPIMEMBER, CUBEMEMBERPROPERTY,
CUBERANKEDMEMBER, CUBESET and CUBESETCOUNT
• IntelliSense style support– In a cell, type “=CU” and all CUBE formulas will display– Select one with arrow keys and hit Tab– When prompted for connection, type " and view pop-up list– Other pop-ups activate on " or "."
At Your Service
• “Range Drag” and relative formula support on CUBEVALUE
• CUBEVALUE and Data Bars go great together
• Ability to convert PivotTables to formulas
CUBExxx Formulas
POWERPIVOT AND EXCEL SERVICES
Self-Service BI with PowerPivot
• Excel + Analysis Services + SharePoint• Enables the working in Excel but mitigates
the “spreadmart” pitfalls:– Use Analysis Services (AS) as a hidden engine
Instead of no engine– Share via SharePoint, accessible by all AS clients
Instead of “deploying” via email– Formal data refresh on server
So data doesn’t get stale, and users don’t have to make effort at updating
– Allow IT to monitorSo it’s not all rogue
– Provide path to more rigorous implementationsCan be upsized to Analysis Services
Column-Oriented Stores• Imagine, instead of:
• You have:
• Perf: values you wish to aggregate are adjacent• Efficiency: great compression from identical or nearly-identical
values in proximity• Fast aggregation and high compression means huge volumes
of data can be stored and processed, in RAM
Employee ID Age Income
1 43 90000
2 38 100000
3 35 100000
Employee ID 1 2 3
Age 43 38 35
Income 90000 100000 100000
Data Import
• Relational databases– SQL Server (including SQL Azure!), Access– Oracle, DB2, Sybase, Informix– Teradata– “Others” (OLE DB, including OLE DB provider for ODBC)
• OData feeds, incl. R2/2012 Reporting Services, Azure DataMarket, WCF Data Services (Astoria), SharePoint 2010 lists, Visual Studio LightSwitch
• Excel via clipboard, linked tables• Filter, preview, friendly names for tables/columns
Calculated Columns and DAX• Formula-based columns may be created• Formula syntax is called DAX (Data Analysis
eXpressions).– Not to be confused with MDX or DMX. Or DACs.
• DAX expressions are similar to Excel formulas– Work with tables and columns; similar to, but distinct from, worksheets
and their columns (and rows)
• =FUNC('table name'[column name])• =FUNCX('table name', <filter expression>)• FILTER(Resellers,[ProductLine] = "Mountain")• RELATED(Products[EnglishProductName])• DAX expressions can be heavily nested
PowerPivot Guidebook
Table tabs
View data in Excel
Calculatedcolumnentry
DAX formula bar
Relationship indicator
Import data fromalmost anywhere
Sort and filter
What’s New? KPIs
Measures
Data and Diagram views
Measuregrid
Measureformula
Sort one column by another
Diagram ViewDefault Aggregations Special Advanced Mode
Reporting
properties
Hierarchies
Hide specific
columns andtables
Create relationshipsvisually
Measures
KPIs
Perspectives
PowerPivot Client
Excel Services
• A component of SharePoint Server 2007/2010; requires Enterprise CAL
• Allows export of workbook, worksheet, or individual items to SharePoint report library– Works great for PivotTables and Charts!– Also for sheets with CUBExxx formulas or conditional
formatting-driven “scorecards”
• Content can be viewed in browser– Excel client not required– Drilldown interactivity maintained– Rendered in pure HTML and JavaScript– Parameterization supported
PowerPivot Server
• Publish to Excel Services• Viewing and interacting• Data Refresh• Treating as SSAS cube
– 2008 R2 version: URL to .xlsx as server name– 2012 version: use POWERPIVOT named instance and treat
just like SSASDb name is GUID-based; best to discover it
– Use Excel, Reporting Services as clientsAnd now Power View too…more later
The IT Dashboard
Increase IT efficiency:Familiar Technologies for Authoring, Sharing, Security, and ComplianceCustomizable IT DashboardVisualize usage with animated charts
Simplify management of SSBI content usingIT Operations Dashboard for SharePoint
PowerPivot Server
Analysis Services Tabular Mode
• SSAS Tabular Mode is the enterprise/server implementation of PowerPivot
• You must have a dedicated tabular mode SSAS instance
• Tabular SSAS projects: BI Developer Studio (BIDS) gone PowerPivot– Implements equivalent tooling to PowerPivot Window– Can create an SSAS tabular database project by
importing an Excel workbook with PowerPivot model
• SSAS tabular models support partitions and roles
SSAS Tabular Project in BIDS
SSAS tabular projectmenus and toolbar
Measure grid and formula bar
Reporting properties in Properties window
DirectQuery Mode
• In DQ mode, model defines schema, but is not used for data
• Queries issued directly against source
• Similar to ROLAP storage for conventional cubes
• Combine with xVelocity ColumnStore indexes for fast, real-time querying
SSAS Tabular Mode
POWER VIEW
What is Power View?
• Ad hoc reporting. Really!• Analysis, data exploration• Data Visualization• In Silverlight, in the browser, in SharePoint• Feels a little like Excel BI• Is actually based on SSRS
– Power View makes a special RDL file
Power View Data Sources
• Power View works only against PowerPivot/SSAS tabular models– DirectQuery mode supported, however
• For PowerPivot, click “Create Power View Report” button or option on workbook in SharePoint PowerPivot Gallery
• For SSAS tabular model, create BISM data source, then click its “Create Power View Report” button or option– BISM data sources can point to PowerPivot
workbooks too, if you want.
Power View!In the browser, in Silverlight
Ribbon, like Excel
Field list, like Excel
Variety of visualizationsand data formats
Data regions pane,like Excel
View Modes
Maximize one chart,fit report to window, put whole report in Reading Mode or Full Screen
Create multiple pages (views)
Power View Basics
Constraining Your DataIn Power View
• Tiles– A filtering mechanism within a visualization
• Highlighting– Selection in one visualization affects the others
• Slicers– Similar to Excel against PowerPivot
• True Filters– Checked drop-down list; very Excel-like– Right-hand filter pane, similar to SSRS and Excel
Services
Power View Filtering
Scatter/Bubble Charts
• Allow for several measures• Features a “play” axis which can be
manipulated through a slider or animated• Excellent way to visualize trends over time
Multipliers
• Multiple charts within a chart, in columns, rows, or a matrix– Horizontal and vertical multipliers
• Allows for visualizing 1 or 2 additional dimensions
Advanced Properties
• Setting the representative column and image tells Power View how to summarize your data, and show stored images
• Other properties tell it about key attributes, default aggregations and more
• Reminder: “DirectQuery” mode tells Power View to get data from relational data source instead of columnar cache– Use with columnstore indexes to have the best of both
worlds– columnstore indexes require Enterprise Edition,
available in BI Edition
Power ViewAdvanced Features
Vocabulary
• MOLAP: Multidimensional OLAP• UDM: Unified Dimensional Model• Cube: unit of schema in a dimensional
database
• xVelocity Columnstore Technology: PowerPivot/SSAS’ column store engine
• VertiPaq: Old name for xVelocity• BISM: BI Semantic Model• IMBI: In-Memory BI engine• Tabular: a column store-based model
– Because it uses tables, not cubes
xVelocity Columnstore Indexes• Implementation of xVelocity columnar technology
engine for SQL Server relational databases– Code name was: “Apollo”
• Use it by creating a columnstore index– CREATE COLUMNSTORE INDEX index ON table (col1, Col2, …)
• Can ignore it in a SELECT, too:– OPTION (IGNORE_NONCLUSTERED_COLUMNSTORE_INDEX)
• Can significantly increase performance of star join queries (i.e. aggregating queries with dimension lookups).
• Must enable “batch” mode as well – look @ query plan to confirm!
• Not as good as SSAS, but better than plain old GROUP BY
REPORTING SERVICES AND REPORT BUILDER
• Author Impactful Reports • Powerful Designers• Flexible Report Layout• Rich Visualizations
• Manage any Workload• Enterprise Scale Platform• Central Deployment • Strong Manageability
• Deliver Personalized Reports• Interactive Reports• Rendering in the Format Users Want• Delivery to Location Users Want
Scalable, Enterprise Reporting Platform
Self-Service Report AuthoringReport Builder 3.0
Familiar Microsoft Office InterfacePowerful WizardsSharePoint list as data source
Powerful Query DesignerFlexible Layout Options w/ rich visualizationsPowerPivot as data source
Rich Visualizations
SSRS + SSAS = YES(S)
• Reporting Services can query Analysis Services databases, including multidimensional cubes and tabular models, directly. PowerPivot too.
• Uses MSOLAP OLE DB provider and issues MDX queries– Has its own query designer for MDX
• Beware: SSRS essentially “flattens” SSAS data– Conforming multidimensional data to relational
structures
Self-Service Reporting?
• Fact: Reporting Services is powerful• Fact: the data visualizations in SSRS are some of
the best in the BI stack• Fact: None of that makes SSRS report design
end-user-friendly• Building SSRS reports, and especially charts,
gauges, etc. from scratch is tedious• Until now, best end-user option has been to copy
an existing report and tweak it. Yech!• There must be a better way…
Report Parts to the Rescue
• Skilled SSRS designers can publish report parts– From Report Builder 3.0 or VS report projects
• End users can pick them from a gallery– A task pane, in Report Builder 3.0, with search capability– Cannot select from VS report designer
• What can be published?:– Tablixes (i.e. tables, matrices)– Rectangles– Images, Charts, Gauges, Maps– Parameters and Lists
• All aided by new ability to share Datasets and original ability to share Data Sources
Easy to Publish; Easy to Select
SSRS Report Parts
PERFORMANCEPOINTSERVICES
PerformancePoint Services (PPS)
Scorecards
Cascading scorecards with interactive charts and data from multiple sources
Analytics
Multi-dimensional slice and dice for advanced analytics including decomposition tree, performance map, and perspective view
PPS Capabilities
• Analytic Grids & Charts– PPS’ own data visualizations– AJAX-based interactive analytics capabilities
• Scorecards– Key Performance Indicators– Objectives– Dimensions
• Dashboards– Bird’s eye view of business activities– Display Analytic Grids and Charts; Scorecards– Integrate SSRS, Excel Services (including PowerPivot) content– Add Filters
PerformancePoint ServicesA Finished Dashboard
MASTER DATA SERVICES,DATA QUALITY SERVICESAND STREAMINSIGHT
MDS: Microsoft’s Master Data Management (MDM) tool
• Examples:– Sales states, countries, currencies, customer types– Customers, products– Think of “lookup tables” or just think of dimensions!– Slowly changing non-transactional entities in your data
• What gets stored:– Schemas– Any hierarchies– The data!
• Other features:– Collections, business rules, security, workflows– Versioning
Other Facts• Result of acquisition of Stratature• v1 is an ASP.NET application; UI is “different”• New in v2 (SQL Server 2012):
• Now Silverlight-based; UI is still “different”• Excel add-in for data entry; creation of entities and attributes• Perform matching with DQS before loading
• Includes .NET and Web Services APIs for reading/writing data and creating/editing models
• Does not integrate with Analysis Services tools even though many of its features and concepts mirror those of dimension designer
• Catalog kept in SQL Server database• Deployment packages can be created, shared and deployed
Objects in MDS
• Models– Entities (like tables or SSAS dimensions)
Attributes (like columns/fields or SSAS attributes)Common attributes are Name and Code
Attribute GroupsUsed to taxonomize attributes within tabs in UI
Members (like rows/records or SSAS members)Hierarchies (like SSAS hierarchies)
Derived or ExplicitCollections (like SSAS named sets)
– Versions– Business rules– Workflows
Data Quality Services
• Data Cleansing Tool• New to SQL Server 2012• Result of Zoomix acquisition• Uses Artificial Intelligence algorithms to
detect invalid data and perform matching (for de-duplication)
• Allows manual intervention, too• Can integrate with MDS and SSIS• Cleaner data = better adoption of your BI
project
DQS Concepts
• Knowledge Bases– Domains
“semantic representation[s] of a type of data in a data field…[contain] a list of trusted values, invalid values, and erroneous data.”
– Mapping
• Data Quality Projects– Cleansing (i.e. correcting)
Validate Using Reference Data Services and Azure DataMarket (or 3rd party providers)
– Matching (i.e. de-duping)– Confidence– Profiling, Monitoring
StreamInsight
• Microsoft’s Complex Event Handling (CEP) Product
• Processes data streams that are fast and high-volume
• Highly parallel C++ code assures low latency, high throughput
• Not based on SQL Server, though that is its “ship vehicle”
• Interesting collaborative potential with BizTalk and SSIS
StreamInsight Concepts
• No UI. All interaction is programmatic.• Based on adapter architecture
– Input and output adapters– Buy or build– Sensors, RFID readers, Web logs, market data streams are
possible event sources
• StreamInsight applications– Streams and events can be queried via LINQ from .NET– Server can run in-process, or shared
StreamInsight v1.2: New in SQL Server 2012
• Resiliency: can take snapshots and restore to that saved state after an outage
• Dev experience: LINQ enhancements– Multiple FROM clauses– Nested types
• Extensibility: user-defined stream operators now supported
• Admin features: server-wide and query-specific perf counters, Windows event logging
• Included with Web and Standard Editions
Hadoop on Windows
• Hadoop is the open source implementation of Google’s MapReduce distributed processing engine
• MS working with Hortonworks to implement it on Windows– A full “distro”– Window Server, Windows Azure DIY, Windows Azure self-
serve cluster provisioning
• Also: ODBC driver for Hive– Works with SSRS, SSAS Tabular, PowerPivot and plain
Excel.
• Also: JavaScript console in the browser• Come to my session on Thursday!
BEYOND RELATIONAL
PART IV
FILESTREAM
• Data explosion accelerating the creation and management of unstructured binary large objects (BLOBs)– Photos– Audio– Video– Email messages– Spreadsheets– Documents– Etc.
The Need To Stream
Two choices
BLOBs And The Database
• varbinary(max) column• Integrated management• Transactional• Simplified programming• Bloats the structured file
groups
BLOBs in the database
• Store them in the database• Store them outside the database, either in the file system
or in a dedicated BLOB store
CustomerId FirstName LastName Picture varbinary(max)
235 'John' 'Doe' 0x3B0E95AE3B2F02292F0B…236 'Sally' 'Smith' 0xF3000EEF293039A2002C…
T-SQL
BLOBs Inside The Database
Two choices
BLOBs And The Database
• varbinary(max) column• Integrated management• Transactional• Simplified programming• Bloats the structured file
groups
• Path references to file system• Separate from the database• Not transactional• Complex programming• Doesn’t interfere with
performance
BLOBs in the database BLOBs outside the database
• Store them in the database• Store them outside the database, either in the file system
or in a dedicated BLOB store
CustomerId FirstName LastName Picture varchar(max)
235 'John' 'Doe' 'C:\files\jdoe.jpg'
236 'Sally' 'Smith' 'C:\files\ssmith.jpg'
T-SQL File I/O
BLOBs Outside The Database
• Transparently store varbinary(max) data in the file system– Declare column as “varbinary(max) FILESTREAM”
• Integrated management– BLOBs are logically part of the database (backup, restore, etc.), but stored
physically separate as a file group mapped to the file system
• Simplified programming– SQL Server transparently links rows in relational tables to BLOBs in the file
system– No developer effort required to maintain references between structured
and unstructured data
• Transactional– SQL Server integrates with the NTFS file system– Database transactions wrap NTFS transactions
• Performant– File system is optimized for streaming
BLOBs Using FILESTREAM
CustomerId FirstName LastName Picture varbinary(max) FILESTREAM
235 'John' 'Doe' 0x3B0E95AE3B2F02292F0B…236 'Sally' 'Smith' 0xF3000EEF293039A2002C…
T-SQL
BLOBs Using FILESTREAM
• Set to one of four levels– Disabled– T-SQL only– T-SQL + file system (local only)– T-SQL + file system (remote)
• Enable it either:– During setup– With SQL Server Configuration
Manager– No way to script with T-SQL, but
VBScript file is available on CodePlex that provides command-line alternative
• Security concern of the Windows administrator
Enabling the Service for FILESTREAM
• Security concern of the SQL Server administrator– Windows and SQL admins must agree!
• Set to one of three levels– Disabled, T-SQL only, T-SQL + file system
• Enable it either:– In SSMS Server Properties dialog
– In T-SQL, with:
EXEC sp_configure filestream_access_level, n RECONFIGURE
Enabling the Instance for FILESTREAM
• CREATE DATABASE PhotoLibrary ON PRIMARY (NAME = PhotoLibrary_data, FILENAME = 'C:\DB\PhotoLibrary_data.mdf'), FILEGROUP FileStreamGroup1 CONTAINS FILESTREAM (NAME = PhotoLibrary_group2, FILENAME = 'C:\DB\Photos') LOG ON (NAME = PhotoLibrary_log, FILENAME = 'C:\DB\PhotoLibrary_log.ldf')
Creating a FILESTREAM Database
• CREATE DATABASE PhotoLibrary ON PRIMARY (NAME = PhotoLibrary_data, FILENAME = 'C:\DB\PhotoLibrary_data.mdf'), FILEGROUP FileStreamGroup1 CONTAINS FILESTREAM (NAME = PhotoLibrary_group2, FILENAME = 'C:\DB\Photos') LOG ON (NAME = PhotoLibrary_log, FILENAME = 'C:\DB\PhotoLibrary_log.ldf')
Creating a FILESTREAM Database
• Table requires ROWGUIDCOL column– Attribute applied to a uniqueidentifier (GUID) column– Must be primary key or have unique constraint– Only one ROWGUIDCOL column is permitted per table
• Define BLOB columns as “varbinary(max) FILESTREAM”– Any number of BLOB columns are permitted per table
Creating FILESTREAM Columns
Getting Started with FILESTREAM
• It is not easy, practical, or efficient to manipulate BLOBs in T-SQL
• Build a streaming client with SqlFileStream– System.Data.SqlTypes.SqlFileStream– Part of System.Data.dll in .NET 3.5 SP1 and higher– Inherits from Stream– Constructor takes logical path and transaction context – Wraps OpenSqlFilestream SQL Server native client API– Consumes no SQL Server memory during processing
Introducing SqlFileStream
• Begin Transaction• INSERT/SELECT row• Retrieve BLOB PathName()• Retrieve
GET_FILESTREAM_TRANSACTION_CONTEXT()• Create and use SqlFileStream• Commit Transaction
The SqlFileStream Recipe
CustomerId FirstName LastName Picture varbinary(max) FILESTREAM
235 'John' 'Doe' 0x3B0E95AE3B2F020B…
236 'Sally' 'Smith' 0xF3000EEF293039A2…
T-SQL SqlFileStream
BLOBs Using SqlFileStream
Using SqlFileStream
• Mirroring/HADR– Not supported with mirroring– Supported with HADR (SQL Server 2012 “AlwaysOn”)
• Transparent Data Encryption (TDE)– Does not encrypt files
• Replication– Supported with some restrictions, see BOL
• Log Shipping– Fully supported– Primary and secondary servers require SQL Server 2008 or higher
• Full-Text Search (FTS)– Fully supported
FILESTREAM Limitations & Considerations
• Database Snapshots– Not supported for FILESTREAM filegroups
• Snapshot Isolation Level– Wasn’t supported in SQL Server 2008, supported in 2008 R2 and 2012
• Local NTFS File System– Requires local NTFS file system– RBS (Remote BLOB Store) API makes SQL Server act as a dedicated
BLOB store
• Security– Requires mixed-mode (integrated) security
• SQL Server Express Edition– Fully supported– Database size limit (4GB in SQL Server 2008, 10GB in 2008 R2 and 2012)
does not include FILESTREAM data
FILESTREAM Limitations & Considerations (cont.)
HIERARCHYID
• Today’s most common form of hierarchical data is XML
• XML support added in SQL Server 2005 is great, if:– You want to store and retrieve an entire hierarchy at one time– The data is consumed in XML by client applications
• Parent-child relationships define rigid hierarchies– Can’t support unlimited breadth and depth
Hierarchical Data Is Not Relational
• Forum and mailing list threads
• Business organization charts
• Content management• Product categories• File/folder management
• FileTable in SQL Server 2012
• Many more…– All typically iterated
recursively
Hierarchical Storage Scenarios
• One table– Each row is linked to its parent
• Works, but has limitations– CTEs help with recursive queries– Still your job to manage updates– Manually maintain structure– Complex to reparent entire sub-trees– Difficult to query– Difficult to control precise ordering of siblings
Traditional Self-Join Approach
• System CLR data type– Extremely compact variable-length format– Does not require SQL CLR to be enabled on the server
• Enables a robust hierarchical structure over a self-joining table– Each row is a node with a unique hierarchyid value– Contains the path in the hierarchy to the node… down to the
sibling ordinal position
• Invoke methods in T-SQL– Efficiently query the hierarchy– Arbitrarily insert, modify, and delete nodes– Reparent entire sub-trees with a single update
Introducing hierarchyid
• GetAncestor• GetDescendant• GetLevel• GetReparentedValue• GetRoot• IsDescendantOf• Parse• ToString
hierarchyid Methods
• Two types of indexes:–Use one, the other, or
both as your needs dictate
• Depth-First–Create a primary key or
unique index
• Breadth-First–Create a composite index
that includes a level column
Depth-First Index
Breadth-First Index
Indexing hierarchyid Columns
hierarchyid
hierarchyid Example
FILETABLE
• Builds on FILESTREAM and hierarchyid• A “semi”-ordinary table that houses a logical file system
– Fixed column schema– Each row represents a “folder” or “file”
– Plus 10 storage attributes columns (e.g., is directory, created, modified, archive bit)
• Windows file/directory management API support– A Windows file share exposes the FileTable– Bi-directional – changes to the table are reflected in the file share and vice versa
Introducing FileTable
Column Name Data Type Description
stream_id uniqueidentifier ROWGUIDCOL
Unique row identifier
file_stream varbinary(max) FILESTREAM
BLOB content (NULL if directory)
name nvarchar(255) Name of file or directory
path_locator hierarchyid Location of file or directory within the file system hierarchy
BLOBs Using FileTable
T-SQL
stream_id name path_locator file_stream
27D8D4AD-D100-39…
'Financials' 0xFF271A3562…
NULL
78F603CC-0460-73…
'ReadMe.docx' 0xFF59345688…
0x3B0E956636AE3B2F020B…
T-SQL SqlFileStream
207D4A96-E854-01…
'Budget.xlsx' 0xFD0011039A…
0xF3F359000EEF293039A2…
Server Machine NameServer Instance FILESTREAM Share NameDatabase NameFileTable Namehierarchyid column defines logical fileand folder paths
varbinary(max) FILESTREAM column holds each file’s contents
FileTable Rows
• Prerequisites at the instance level– FILESTREAM must be enabled for the instance
• Prerequisites at the database level– Enable non-transactional FILESTREAM access for the
database (is still transactional internally)– Set a root directory name for all FileTables in the database (this
will become a child in the Windows file share)– Use this T-SQL statement:
CREATE DATABASE … WITH – or – ALTER DATABASE … SET
– …followed by… FILESTREAM( NON_TRANSACTED_ACCESS=FULL|READ, DIRECTORY_NAME='DatabaseRootDirectory')
FileTable Prerequisites
• Create a FileTable in T-SQL– CREATE TABLE TableName AS FileTable
• FileTable has a fixed schema– You don’t (can’t) supply a column list
• Creates logical directory– Logical root directory for the FileTable– Created beneath the root directory for the database– Named after the table, can override by specifying: WITH(FileTable_Directory='TableRootDirectory')
– Exposes a functional Win32 file system– Does not support memory-mapped files (does not
affect remote clients)
Creating a FileTable
GEO-SPATIAL TYPES
• Integrate location awareness into any application– Long been the domain of sophisticated GIS applications
• GIS– A system for capturing, storing, analyzing, and managing data and
associated attributes which are spatially referenced to the earth
• Allow a user to interact with information that is relevant to locations that they care about:– Home, work, school, or vacation destinations
• Two geospatial models– Planar– Geodetic
SQL Server Spaces Out
• Two spatial models = Two system CLR types• geometry
– Planar (flat) model– Flat 2-dimensional Cartesian Coordinate system– X and Y coordinates with uniform units of measure– Use for mapping small areas
• geography– Geodetic (round-earth) model– Latitude and longitude– Use for larger mapping where land mass is too big to fit on
one planar projection
Spatial Data Types
• Two-Dimensional Surface– X and Y coordinates on an arbitrary plane
• Flat Earth Projection– To work with geospatial data on a 2D surface, a projection is
created to flatten the geographical objects on the spheroid– Example: Planar Model based on Mercator Projection
Antarctica
Greenland
North America
Africa
Square KM:- Antarctica = 13 million- Greenland = 2 million- N. America = 24 million- Africa = 30 million
Planar Spatial Model
• Accurate geographic measurements– Locations on planet surface described by latitude and
longitude angles
• Ellipsoidal sphere– Latitude = angle N/S of the equator– Longitude = angle E/W of the Prime Meridian
Geodetic Spatial Model
• Open Geospatial Consortium (OGC)– International standards body
• Microsoft belongs to the OGC– SQL Server 2008 uses the OGC’s Simple Feature Access
standards
• OpenGIS Simple Feature Interface Standards (SFS)– A well-defined way for applications to store and access spatial
data in relational databases– Described using vector elements; such as points, lines and
polygons
• Three ways to import geospatial data– Well-Known Text (WKT)– Well-Known Binary (WKB)– Geographic Markup Language (GML)
Spatial Data Standards
• WKT examples:
• POINT(6 10)• POINT(-111.06687 45.01188)• LINESTRING(3 4,10 50,20 25)• POLYGON((1 1,5 1,5 5,1 5,1 1),(2 2, 3 2, 3 3, 2 3,2 2))• POLYGON((
-75.17031 39.95601, -75.16786 39.95778,-75.17921 39.96874, -75.18441 39.96512,-75.17031 39.95601))
• MULTIPOINT(3.5 5.6,4.8 10.5)• MULTILINESTRING((3 4,10 50,20 25),(-5 -8,-10 -8,-15 -
4))• GEOMETRYCOLLECTION(POINT(4 6),LINESTRING(4 6,7 10))• CIRCULARSTRING(1 5, 6 2, 7 3)
Well-Known Text (WKT)
• STArea• STBuffer• STCentroid• STDifference• STDimension• STDistance• STEnvelope• STGeomFromText• STIntersection• STIntersects• STPointFromText, STLineFromText, STPolyFromText• STPointFromWKB, STLineFromWKB, STPolyFromWKB• STSymDifference• STUnion• GeomFromGml• Parse• ToString• and more (about 70 total)
Geospatial Methods
Geo-Spatial Types
0,0 150,0
150,1500,150
300,150
300,0
150,300 300,300
50,50
100,100
20,180 180,180
• Circular Arcs• CircularString• CompoundCurve• CurvePolygon
• All existing methods work on circular objects• New spatial methods
• BufferWithCurves• STNumCurves, STCurveN• STCurveToLine• CurveToLineWithTolerance• IsValidDetailed• HasZ, HasM, AsBinaryZM• ShortestLineTo• UnionAggregate, EnvelopeAggregate, CollectionAggregate,
ConvexHullAggregate• MinDbCompatibilityLevel
Spatial Improvements InSQL Server 2012
• Improved Precision• Constructions and relations use 48 bits of precision (previously 27
bits)
• geography Enhancements• Support for objects larger than a logical hemisphere (“FullGlobe”)• Support for new and previous “geometry-only” methods
• New SRID• Spatial reference ID 104001 (sphere of radius 1)
• Performance Improvements• Better tuning and hints• Auto Grid indexing with 8 levels (previously 4 levels)
• Other Improvements• New histogram stored procedures• Support for persisted computed columns
Spatial Improvements InSQL Server 2012
• Workshop slides and code– http://sdrv.ms/VSLiveNY2012SQL
• Aaron Bertrand’s Blog– http://sqlblog.com/blogs/aaron_bertrand
• Bob Beauchemin’s Blog– http://sqlskills.com/BLOGS/BOBB
• Itzik Ben-Gan’s Web Site– http://www.sql.co.il
• James Serra’s Blog– http://www.jamesserra.com
• simple-talk – Learn SQL Server– http://www.simple-talk.com/sql/learn-sql-server
Resources
• Contact us– [email protected]– [email protected]
• Visit our blogs– brustblog.com – lennilobel.wordpress.com
• Follow us on Twitter– @andrewbrust– @lennilobel
• Thanks for coming!
Thank You!