biml: bi to the next level

13.12.2014

BI Markup Language:

BI to the next level

Davide Mauri

SolidQ – Founder & Mentor

@mauridb

www.solidq.com

dmauri@solidq.com

13.12.2014

Thank you to all our SPONSORS!

13.12.2014

Davide Mauri

20 Years of experience on the SQL Server Platform Specialized in Data Solution Architecture, Database Design,

Performance Tuning, Business Intelligence

Projects, Consulting, Mentoring & Training Regular Speaker @ SQL Server events

Microsoft SQL Server MVP

President of UGISS (Italian SQL Server UG)

Mentor @ SolidQ

Find me here: Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

Twitter: @mauridb

13.12.2014

Agenda

BI, DWH & ETL today

BIML – What’s that?

Tools

BIDS Helper

MIST

BIML Kick Off

Basic Concepts

Automation With “Embedded” C#

BIML & PowerShell

13.12.2014

Building a BI/DWH/ETL in 2014

Is still a (almost) manual process

A *lot* of repetitive low-value work

No (or very few) standard tools available

13.12.2014

How it should be

Semi-automatic process

“develop by intent”

Define the mapping logic from a semantic

perspective

Source to Dimensions / Measures

(Metadata anyone?)

Design the model and let the tool build it for you

13.12.2014

How it should be

Semi-automatic process

“develop by intent”

Define the mapping logic from a semantic

perspective

Source to Dimensions / Measures

(Metadata anyone?)

Design the model and let the tool build it for you

CREATE DIMENSION Customer

FROM SourceCustomerTable

MAP USING CustomerMetadata

ALTER DIMENSION Customers

ADD ATTRIBUTE LoyaltyLevel

AS TYPE 1

CREATE FACT Orders

FROM SourceOrdersTable

MAP USING OrdersMetadata

ALTER FACT Orders

ADD DIMENSION Customer

13.12.2014

The perfect BI process & architecture

Iterative!

13.12.2014

Design Pattern

“A general reusable

solution to a

commonly occurring

problem within a

given context”

13.12.2014

No Monkey Work!

Let the people think

and let the machines

do the «monkey»

work.

13.12.2014

BIML – What’s that?

Business Intelligence Markup Language

XML-Based

BI Object-Generator

SQL Server

Integration Services

Analysis Services

13.12.2014

BIML – What’s that?

Allows a declarative approach to define BI

objects

Favor automation

Human-readable

Compatible with SCC system

Allows merge of files

Born from “Project Vulcan”

Still available on CodePlex

13.12.2014

BIML – What’s that?

Once objects are generated, BIML is not needed

anymore

No additional runtime/components required

Support for SQL Server 2008 & 2012 objects

2014 is coming very soon

Anyway generated packages are automatically

converted into 2014 packages by Visual Studio 2013,

without problems

13.12.2014

BIML – What’s that?

BIMLScript is based on the T4 templating engine.

Allows the generation of BIML using

C#

VB.NET

BIML + BIMLScript = HTML + ASP Classic

13.12.2014

Tools

BIDS Helper

http://bidshelper.codeplex.com/

Visual Studio Plugin

Free

Source Code Available

Not all BIML features are supported

“Limited” to SSIS generation only

13.12.2014

Tools

MIST

Full-Featured BIML IDE

Buy or Rent

Made by BIML creator: Varigence

13.12.2014

DEMO

First Contact with BIML

13.12.2014

Basic Concepts

A single BIML file can generate more than one

SSIS Package

All SSIS object and related properties are exposed

via BIML elements

Additional specific attribute: ConstraintMode

Linear or Parallel

13.12.2014

Basic Concepts

Parallel Constraint Mode

“Inputs” connected to “Outputs”

SSIS Variables: “dot” notation

13.12.2014

Basic Concepts

Control Flow and Data Flow 100% supported

In Data Flow it’s mandatory to define input and

outputs

Since they aren’t constraints, but data streams

13.12.2014

DEMO

BIML is your friend!

13.12.2014

Automation with BIMLScript

BIML Script (T4) Directives: <#@ and #>

13.12.2014

Automation with BIMLScript

BIML Script Tags: <# e #>

Allows usage of C# / VB.Net within BIML

The “=” symbol allows “in-line” code injection

13.12.2014

Automation with BIMLScript

Metadata is needed in order to define what to

generate

Metadata can be extracted by

BIML Engine (Hadron)

T-SQL Queries

13.12.2014

Automation with BIMLScript

Metadata via BIML Engine (Hadron)

13.12.2014

Automation with BIMLScript

Metadata via SQL Query

13.12.2014

DEMO

BIMLScript RULES!

13.12.2014

Debugging BIML

With complex BIML Script it can be difficult to

“see” what’s happening behind the scenes

With MIST you can see the resulting BIML

generated from BIML Script

But with BIDS Helper you’re on your own.

NLog can help a lot here!

13.12.2014

Debugging BIML

Nlog: free, powerful & flexible logging infrastructure

http://nlog-project.org/

Stand-Alone Assembly

Just reference the assembly in the BIML Script (T4) template and then use Nlog objects & methods

http://sqlblog.com/blogs/davide_mauri/default.aspx

13.12.2014

DEMO

BIMLScript & NLog

13.12.2014

BIML & PowerShell

What if you don’t like C#...

Or if BIMLScript looks to complex?

Especially without MIST?

One option is to generate BIML using PowerShell

Is just XML after all….

13.12.2014

DEMO

BIML & PowerShell

13.12.2014

Conclusions

Use BIML!!!!

It helps to

Make better use of your time

Create your own ETL patterns

http://www.slideshare.net/davidemauri/automating-dwh-

patterns-through-metadata

Support Refactoring & Enables Agility

Favor use of metadata

13.12.2014

References

BIML Script

http://bimlscript.com/

Stairway to BIML

http://www.sqlservercentral.com/stairway/100550/

Automating DWH Patterns Through Metadata

http://www.slideshare.net/davidemauri/automating-

dwh-patterns-through-metadata

13.12.2014

Q&A

Questions?

13.12.2014

Thanks!