tools and tips: from accidental to efficient data warehouse developer (sqlsaturday portugal)
TRANSCRIPT
Tools and Tips:From Accidental to EfficientData Warehouse Developer
Cathrine Wilhelmsen - SQLSaturday Lisbon 2015
Session descriptionYou have probably heard about the Accidental DBA, but what about the Accidental Data Warehouse developer? We stumbled into the world of data warehousing, learned dimensional modeling and work with T-SQL and SSIS daily. We're masters of googling solutions to our problems and make sure our complex ETL processes run without errors. We deliver data to business users... but we don't deliver data as fast as we want.
You might not be able to rewrite your entire data warehouse or change your team's processes over night, but there are many things you can do to increase your own productivity and become a more efficient data warehouse developer.
In this session I will show you some of what I've learned and discovered that has made me burst out "Oh wow! Why did I not know this yesterday!?" including query improvements, free tools and scripts, SSMS features and even a couple of things I used to think were only useful for those scary DBAs.
Say thank you to organizers and volunteers!
They spend their FREE time to give you this event
Because they are crazy
Because they want YOU to learn from the BEST IN THE WORLD
Important Activities
Sponsor Sessions with Raffles
• 15:10 - Rumos, BI4ALL, Bold Int, CozyRoc, Pythian
WIT (Women in Technology)
• 15:10 at BizSpark Room (Ground Floor)
SQLClinic Challenges
• 17:00 BI (Steph Locke)
Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.netData Warehouse Architect
Business Intelligence Developer
Oh, by the way...
SELECT SUM(rows)
FROM sys.partitions
WHERE index_id IN (0, 1)
AND object_id =OBJECT_ID('Sales');
...querying sys.partitions can be better and faster than COUNT(*)
Keyboard Shortcuts
Assign shortcuts you frequently use
Remove shortcuts you accidentally click (no more "ooops")
msdn.microsoft.com/en-us/library/ms174205.aspx
Free Tool: Redgate SQL Search
red-gate.com/products/sql-development/sql-search/
Advanced Snippets and Formatting
Redgate SQL Prompt (Licensed)
ApexSQL Complete / Refactor
SSMS Tools Pack (Licensed)
SSMS Boost
Poor Man's T-SQL Formatter
dbForge SQL Complete (Licensed)
red-gate.com
apexsql.com
ssmstoolspack.com
ssmsboost.com
poorsql.com
devart.com/dbforge
Registered Servers
Save and group servers
Is the server running?
Multiserver Queries
View Registered Servers
CTRL ALT G
SARGable queries
"The query can efficiently seek using an index to find the correct rows searched for in WHERE or JOIN clauses"
Compare it to finding a person in a phone book
(We'll pretend we still use phone books)
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
SARGable queries
Find all rows where Name starts with "T"
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
SARGable queries
Find all rows where Name starts with "T"
Non-SARGable queries
"The query has to scan each row in the table to find the correct rows searched for in WHERE or JOIN clauses"
Compare it to finding a person in a phone book
(We'll keep pretending we still use phone books)
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
Non-SARGable queries
Find all rows where Name contains "al"
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
Non-SARGable queries
Find all rows where Name contains "al"
WHERE CAST(EpisodeDate AS DATE) = '20050114'
WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'
WHERE YEAR(EpisodeDate) = 2005
WHERE EpisodeDate >= '20050101'AND EpisodeDate < '20060101'
SARGable or Non-SARGable?
WHERE CAST(EpisodeDate AS DATE) = '20050114'
WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'
WHERE YEAR(EpisodeDate) = 2005
WHERE EpisodeDate >= '20050101'AND EpisodeDate < '20060101'
SARGable or Non-SARGable?
WHERE Survivors < 40000
WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000
WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000
SARGable or Non-SARGable?
WHERE Survivors < 40000
WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000
WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000
SARGable or Non-SARGable?
sqlbits.com/Sessions/Event7/Understanding_SARGability_to_make_your_queries_run_faster
Free Tool: SQL Sentry Plan Explorer
sqlsentry.com/products/plan-explorer
Free Book: SQL Server Execution Plansby Grant Fritchey
red-gate.com/community/books
Free Script: sp_WhoIsActive by Adam Machanic
sqlblog.com/blogs/adam_machanic
Business Intelligence Markup Language
Easy to read and write XML dialect
Generate SSIS packages from metadata
What do I need?
Free add-in for BIDS / SSDT-BIbidshelper.codeplex.com
Biml syntax<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<OleDbConnection Name="Source" ConnectionString="…" />
</Connections>
<Packages>
<Package Name="EmptyPackage">
…
</Package>
</Packages>
</Biml>
The magic is in the
Extend Biml with C# or VB.NET code blocks
Import database structure and metadata
Loop over tables and columns
Add expressions to replace static values
BimlScript syntax
<#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #>
<# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "..."); #>
<# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #>
<Packages>
<# foreach (var table in AW2014DB.TableNodes) { #>
<Package Name="Load_<#=table.Name#>">
…
</Package>
<# } #>
</Packages>
cathrinewilhelmsen.net/efficient
Not enough details? Too fast? Don't worry!Slide deck, links and resources:
Thank you! Enjoy your lunch
@cathrinew
cathrinewilhelmsen.net
no.linkedin.com/in/cathrinewilhelmsen
cathrinewilhelmsen.net/efficient