biml: bi to the next level
TRANSCRIPT
13.12.2014
BI Markup Language:
BI to the next level
Davide Mauri
SolidQ – Founder & Mentor
@mauridb
www.solidq.com
13.12.2014
Thank you to all our SPONSORS!
13.12.2014
Davide Mauri
20 Years of experience on the SQL Server Platform Specialized in Data Solution Architecture, Database Design,
Performance Tuning, Business Intelligence
Projects, Consulting, Mentoring & Training Regular Speaker @ SQL Server events
Microsoft SQL Server MVP
President of UGISS (Italian SQL Server UG)
Mentor @ SolidQ
Find me here: Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx
Twitter: @mauridb
13.12.2014
Agenda
BI, DWH & ETL today
BIML – What’s that?
Tools
BIDS Helper
MIST
BIML Kick Off
Basic Concepts
Automation With “Embedded” C#
BIML & PowerShell
13.12.2014
Building a BI/DWH/ETL in 2014
Is still a (almost) manual process
A *lot* of repetitive low-value work
No (or very few) standard tools available
13.12.2014
How it should be
Semi-automatic process
“develop by intent”
Define the mapping logic from a semantic
perspective
Source to Dimensions / Measures
(Metadata anyone?)
Design the model and let the tool build it for you
13.12.2014
How it should be
Semi-automatic process
“develop by intent”
Define the mapping logic from a semantic
perspective
Source to Dimensions / Measures
(Metadata anyone?)
Design the model and let the tool build it for you
CREATE DIMENSION Customer
FROM SourceCustomerTable
MAP USING CustomerMetadata
ALTER DIMENSION Customers
ADD ATTRIBUTE LoyaltyLevel
AS TYPE 1
CREATE FACT Orders
FROM SourceOrdersTable
MAP USING OrdersMetadata
ALTER FACT Orders
ADD DIMENSION Customer
13.12.2014
The perfect BI process & architecture
Iterative!
13.12.2014
Design Pattern
“A general reusable
solution to a
commonly occurring
problem within a
given context”
13.12.2014
No Monkey Work!
Let the people think
and let the machines
do the «monkey»
work.
13.12.2014
BIML – What’s that?
Business Intelligence Markup Language
XML-Based
BI Object-Generator
SQL Server
Integration Services
Analysis Services
13.12.2014
BIML – What’s that?
Allows a declarative approach to define BI
objects
Favor automation
Human-readable
Compatible with SCC system
Allows merge of files
Born from “Project Vulcan”
Still available on CodePlex
13.12.2014
BIML – What’s that?
Once objects are generated, BIML is not needed
anymore
No additional runtime/components required
Support for SQL Server 2008 & 2012 objects
2014 is coming very soon
Anyway generated packages are automatically
converted into 2014 packages by Visual Studio 2013,
without problems
13.12.2014
BIML – What’s that?
BIMLScript is based on the T4 templating engine.
Allows the generation of BIML using
C#
VB.NET
BIML + BIMLScript = HTML + ASP Classic
13.12.2014
Tools
BIDS Helper
http://bidshelper.codeplex.com/
Visual Studio Plugin
Free
Source Code Available
Not all BIML features are supported
“Limited” to SSIS generation only
13.12.2014
Tools
MIST
Full-Featured BIML IDE
Buy or Rent
Made by BIML creator: Varigence
13.12.2014
DEMO
First Contact with BIML
13.12.2014
Basic Concepts
A single BIML file can generate more than one
SSIS Package
All SSIS object and related properties are exposed
via BIML elements
Additional specific attribute: ConstraintMode
Linear or Parallel
13.12.2014
Basic Concepts
Parallel Constraint Mode
“Inputs” connected to “Outputs”
SSIS Variables: “dot” notation
13.12.2014
Basic Concepts
Control Flow and Data Flow 100% supported
In Data Flow it’s mandatory to define input and
outputs
Since they aren’t constraints, but data streams
13.12.2014
DEMO
BIML is your friend!
13.12.2014
Automation with BIMLScript
BIML Script (T4) Directives: <#@ and #>
13.12.2014
Automation with BIMLScript
BIML Script Tags: <# e #>
Allows usage of C# / VB.Net within BIML
The “=” symbol allows “in-line” code injection
13.12.2014
Automation with BIMLScript
Metadata is needed in order to define what to
generate
Metadata can be extracted by
BIML Engine (Hadron)
T-SQL Queries
13.12.2014
Automation with BIMLScript
Metadata via BIML Engine (Hadron)
13.12.2014
Automation with BIMLScript
Metadata via SQL Query
13.12.2014
DEMO
BIMLScript RULES!
13.12.2014
Debugging BIML
With complex BIML Script it can be difficult to
“see” what’s happening behind the scenes
With MIST you can see the resulting BIML
generated from BIML Script
But with BIDS Helper you’re on your own.
NLog can help a lot here!
13.12.2014
Debugging BIML
Nlog: free, powerful & flexible logging infrastructure
http://nlog-project.org/
Stand-Alone Assembly
Just reference the assembly in the BIML Script (T4) template and then use Nlog objects & methods
http://sqlblog.com/blogs/davide_mauri/default.aspx
13.12.2014
DEMO
BIMLScript & NLog
13.12.2014
BIML & PowerShell
What if you don’t like C#...
Or if BIMLScript looks to complex?
Especially without MIST?
One option is to generate BIML using PowerShell
Is just XML after all….
13.12.2014
DEMO
BIML & PowerShell
13.12.2014
Conclusions
Use BIML!!!!
It helps to
Make better use of your time
Create your own ETL patterns
http://www.slideshare.net/davidemauri/automating-dwh-
patterns-through-metadata
Support Refactoring & Enables Agility
Favor use of metadata
13.12.2014
References
BIML Script
http://bimlscript.com/
Stairway to BIML
http://www.sqlservercentral.com/stairway/100550/
Automating DWH Patterns Through Metadata
http://www.slideshare.net/davidemauri/automating-
dwh-patterns-through-metadata
13.12.2014
Q&A
Questions?
13.12.2014
Thanks!