SSIS Field Notes

Darren GreenKonesans Ltd

My Checklist

• Standards• Logging• Configuration

Common Princiapls

• Common design decisions or patterns• Logging• Frameworks• Custom vs Stock

Basic Standards

• Solution and Project structure• ETL vs ELT• Staging• Custom components

Source Control

• Products– Team Foundation Server– Subversion (Visual SVN)– Others…

• Issues– Cannot merge or use standard conflict resolution

• BIDS Helper- Smart Diff– TFS, SourceSafe,File

• BI Smart Diff– Subversion

Naming Conventions

• Prefix notation, e.g. DFT for Data Flow Task– http://consultingblogs.emc.com/jamiethomson/

• Expand Name property– SQL Create Year Staging Table

• Expand Description property– Create the year named staging table. Any existing

table will be dropped first.• Documentation tools are not clever

Logging

• Performance monitoring– Real-time monitoring– Trending to justify upgrades• Re-write problem packages• Upgrade hardware / network / environment

• Problem solving– Why did the job fail last night?

Logging Options

• Built in SSIS logging– Good for standard stuff, including errors

• Maintenance routine to prune records• Delete Info daily, Warnings weekly, Errors monthly

• Custom SSIS logging– Log process specific metrics - row counts– Event Handler or Control Flow

• Windows Logging– Event Log– Performance Monitor

Prune – Delete in ChunksSET @Count = 1000WHILE @Count IS NOT NULL AND @Count > 0BEGIN DELETE TOP (@Count) FROM dbo.sysdtslog90 WHERE StartTime < @MinStartTime SET @Count = @@ROWCOUNT

-- Pause for 0.2 seconds WAITFOR DELAY '000:00:00.200'END

Frameworks

• Consistent logging approach• Process or package state– Passing state between package processes– Saving state for the next run• Last extract date• Complete run only once per day

• Dynamic execution workflow– Managed in tables not Control Flow• Easier to manage logical units and reuse

Frameworks

• Standard approach reduces costs– Lower support costs– Higher quality through reuse

• Cost of applying and maintaining framework– Maintainable frameworks are key!– Custom components encapsulate code– Use the API to bulk apply changes

Custom Components

• Task• Pipeline Component– Source– Destination– Transformation

• Log Provider• Connection Manager• For Each Enumerators

Using Custom Components?

• Easy to manage and update– Good for re-use– Can add/edit functionality easily – One file per machine for all packages– Good for frameworks or complex operations– Good debugging and testing support

• Require .NET development skills• External dependency– Additional step during the initial deployment, but also

single update step thereafter

Stock Components & Scripting

• Faster to develop• Familiar and easy to understand– Don’t write your own data flow engine in a script

component!– Acknowledge the need for reuse when it exists and

create a shared external assembly • Self contained - No external dependency• Copy and paste package maintenance– Not good for frameworks or common patterns

Recovery & Restarts

• Checkpoints – Native• CheckpointFileName, SaveCheckpoints,

FailPackageOnFailure

– Task level restart only– Partition your Data Flow• Raw files

– Variable values are persisted– Configurations not refreshed– Event handlers within checkpoint scope

Recovery & Restarts

• Auto-Recovery – Roll your own– Check with IF EXISTS…– Framework workflow• Table of packages with status• Package or task level restart

– Variables and precedence constrains expressions

– Delete and re-load• No change tracking or updates

Wake up – a quick demo…

Anti-Pattern #1

• Send Mail Task for Errors– Control Flow or Event Handlers– Exceptions cause multiple events– Use external host to handle exceptions• SQL Agent

– Use a report

Anti-Pattern # 2

• Multiple Configuration Types– Use one configuration type• Use indirect or ordered pointer configuration

– Easier for support and deployment

Sorting

• Why sort?– Merge Join

– Ordered Analysis• Running totals• Ranking• Script Component• Custom Component • Rank Transform

Sorting

• How to sort?– Sort Transformation– Ordinal Nsort– Ivolva ExtraSort (2005)– Sort in Source – ORDER BY

Sorting

Thank you

• Questions?

• Darren Green– darren@konesans.com– www.konesans.com