scaling agile data warehousing with the scaled agile framework (safe)
TRANSCRIPT
Em Campbell-Pretty Partner, Context Matters @PrettyAgile www.prettyagile.com au.linkedin.com/in/ejcampbellpretty/ [email protected]
Scaling Agile Data Warehousing with SAFe
Agile Denver Meetup 8th October 2015
THE ENTERPRISE DATA WAREHOUSE
80 TB of business data
28 Teradata nodes 3 TB of new data per month
350 SQL users 47 business systems feeding into the EDW
400 million records loaded daily 13 billion records extracted daily
43 billion records in largest table 4 tables with over 35 billion records
124 different client apps
6.8 million queries daily 16K tables, 28K views
and 2K macros 7K jobs running daily
21K processes
THE EDW DELIVERY TEAM
• Amazon search
AGILE DATA WAREHOUSING?
WIDE PIPES
Image Source: http://ivyleagueinsecurities.com/2010/07/missing-pieces/
Image Source: http://phandroid.com/2011/05/24/unlock-your-samsung-galaxy-s-2-for-use-on-att-or-t-mobile/
Photo (https://flic.kr/p/FryLL) by Harold Walker CC BY-NC 2.0
THE FIRST “AGILE” TEAM
FEBRUARY
MAY
AUGUST
OCTOBER
NOVEMBER
SCALING “WAGILE”
Photo Source: https://flic.kr/p/ndZRQj
Project Manager
Product Owner Business Analyst Change Lead Technical Lead
Logical Data Modeller
Physical Data Modeller
Physical Data Modeller
Vender Project Manger
On Site ETL On Site ETL
On Site ETL Off shore ETL
Off shore ETL Off shore ETL
DBA
Business Intelligence Test Lead
Test Analyst
Enterprise Architect Data SME
Scrum Master
A 22 PERSON AGILE TEAM???
Team Size
Throughput
Scrum Master
Technical Lead Test Lead Developer Developer Developer Developer Developer
SMALL CROSS FUNCTION TEAMS
The Agile Release Train
Agile Technical Practices
Photo scoured from: https://flic.kr/p/8ejM1e
GM
EDW Program Solution Design & Build BAU Program
SHAPING THE EDW AGILE RELEASE TRAIN
Transition Services
SHAPING THE EDW AGILE RELEASE TRAIN
Deployment Services
Development Services
Pipeline Services
SET THE DATE
WHAT ABOUT PROJECTS?
VISUALIZE EVERYTHING
EPIC OWNER
FEATURE OWNER
PRODUCT OWNER
CONTENT AUTHORITY
SCALING DOWN FROM 18 PROJECT MANAGERS
FUNDING THE AGILE RELEASE TRAIN (ART)
COST OF ART ---------------------- ART VELOCITY
COST PER STORY POINT =
PREPARING THE PSI BACKLOG
8 week PSI
8 week PSI 6 week PSI
8 week PSI 6 week PSI 4 week PSI
8 week PSI 6 week PSI 4 week PSI
2 week PSI????
UNITY DAY (PSI-LITE)
BAKE IN TIME TO IMPROVE
KEEPING THE TRAIN ON THE TRACKS
COMMUNICATE, COMMUNICATE, COMMUNICATE
9:00am 9:15am 930am
9:45am
BUBBLE UP ISSUES OUTSIDE TEAM CONTROL
USE SPECIALIST CHAPTERS
APPLY LEAN LEADERSHIP
REINFORCE TEAM & ART IDENTITY
World Leaders in Agile Data Warehousing
IMPLEMENT EXTREME PROGRAMMING
INVEST IN ENABLING INFRASTRUCTURE
A LITTLE TASTE OF OUR CULTURE
A LITTLE TASTE OF OUR CULTURE
ü Average delivery cycle time down from 12 month to 3 months ü Frequency of deployment increased from quarterly to weekly ü Cost to deliver down 70% ü 95% decrease in product defects ü 99% projects delivered on time and on budget ü Happy project sponsors (NPS +50) ü Happy teams (NPS +56)
THE RESULTS
“This is probably the most well-scaled agile data warehousing group I’ve ever observed, and they are still getting better”
Ken Collier Director of Agile Analytics at Thoughtworks & author of Agile Analytics
Em Campbell-Pretty Partner, Context Matters @PrettyAgile www.prettyagile.com au.linkedin.com/in/ejcampbellpretty/ [email protected]
QUESTIONS?
WANT TO KNOW MORE? Check out my blog: www.prettyagile.com
You can also download a free copy of my article on “Scaling Agile Data Warehousing” from the Cutter IT Journal using this link: http://www.cutter.com/offers/campbellpretty.html