empiricism and guessing among oracle professionals jeremiah wilton
TRANSCRIPT
Empiricism and GuessingEmpiricism and GuessingAmong Oracle ProfessionalsAmong Oracle Professionals
Jeremiah Wiltonhttp://www.ora-600.net
ORA-600 Consulting
Independent Consulting Multiple clients Emergencies Seminars Recruiting
Me and Guessing
Started out guessing a lot Felt my work was sound Insufficient mentoring/oversight No overall method to draw from No tradition of excellence
Oracle and Guessing
7.3 Tuning Guide is a manual in guessology Where are wait events described? Emphasis on long lists of stuff
What sounds good? BCHR – OK “Spin Count” - Cool!
Seemed more art than science
Erroneous Literature
Literature was full of nonsense – written by mischievous elves
Still 80% correct Would have been helpful but I could get by
without it so I mainly did
Compensation Issue
One of my problems was compensation Late '90s steep increase in DBA salaries Low pay = low motivation
Amazon and Me
Wild guesses and misinformation fell flat High expectations for accuracy and knowledge Gifted colleagues set level of excellence Better compensation Afraid to let colleagues down
Sad state of our profession
Poor quality is common Unqualified DBAs are often well compensated Driven by law of supply and demand Insufficient resources for poor DBAs get better Oracle is specialized and complex and it is easy
to fool people
Role of hiring in low quality
Non-technical/non-DBAs hiring DBAs Unqualified DBAs get hired at good salaries Creates resentment among colleagues Waste of money No valid review process without senior technical
leadership Hire someone to help do this right!
You do you know you are unqualified?
Poor DBAs don't know there's a problem Manual? What manual? No mentoring Only exceptional people can self-develop in
these circumstances
The boys' club
Personal circumstances play into work hours and availability
80-hour DBAs can be really good 40-hour should be equally acceptable Freebie for company Encourages low DBA headcount Emphasis should be on quality Reasonable work levels ≠ unqualified
More opportunities for improvement
IOUG.org OTN Oracle-L comp.databases.oracle.server
Has candidate ever posted to a forum? Goes to motivation, interest in learning or
interest in sharing knowledge
State of DBA candidates
So many unqualified candidates Actually working! Unbelievable!
TABOS
Ask any DBA candidate the “slow database” question
Answer is usually a punchlist in no particular order
TABOS = Try A Bunch Of Stuff Oracle Support uses TABOS
Origins of TABOS in Oracle docs
Oracle 7 Tuning Guide Insufficient CPU Insufficient memory Insufficient I/O Network constraints Software constraints
Pointed DBAs at v$sysstat Scant and muddled references to wait events
Candidates echo the manual
Candidate espousing TABOS sounds like the Oracle7 Tuning Guide
Check CPU usage
Check memory usage
Check for high BCHR
Check for some other hit/miss ratios you found on the Internet
Check for full disks
Check for too many extents
Check for “fragmentation”
Check for disk sorts
Check for locks
Check for swapping
Check for FT scans
Check for bad plans
etc...
TABOS... works?
You might stumble upon the problem Works enough to just keep practitioners
employed Not fast or clever, but... Sounds “technical” - lots of jargon Makes you look busy! But might never find the problem if not on your
list Waste of time
Is TABOS for real?
Just a figment of the job interview Even underqualified DBAs use some
experience to order their list of checks Experience trumps a wild guess I wouldn't use even experience-driven TABOS Using waits and ASH still faster and more
reliable
Good Guesses
Some educated guesses are effective All DBAs guess Educated guess is as good as the education Empirical from εμπειρισμός = experience Experience is a valid form of evidence/data
Pure Empiricism
Are there purely empirical DBAs? Yes: BAAG – Battle Against Any Guess
http://www.battleagainstanyguess.com “Eliminate guesswork from the decision-making
process” BAAG in the tradition of BAARF
BAAG has a point
Oracle's instrumentation is the best way to solve most problems
“Slow database” question is the poster child for ASH
Starting with ASH/waits assumes that Oracle is the problem
For reality-based troubleshooting, we need equally good instrumentation on app server
How empirical can you really be?
Time and resources limit empiricism Many decisions don't merit exhaustive means A good DBA knows when to make an educated
guess
The Axis of Guessing
Empiricism TABOSEducatedGuessing
Most DBAs
There are no pure empiricists
So why BAAG? An effort to distance from messy thought Large number of unqualified and undertrained
DBAs Polarizing effect on skilled DBAs Real axis is learners vs. slackers
A meaningless axis
Separatism is destructive Us vs. Them Imaginary division Anti-guessing ≠ Science The key to a scientific approach is verification Oracle's instrumentation is ample
Issue of incentive
Most DBAs won't improve without incentive High DBA demand reduces incentive Non-DBAs can't tell a DBA is unqualified
Manuals are the first step
Oracle7 Tuning Guide notwithstanding Manuals are pretty good now Require patience and persistence Some people are not book learners “Experiential learners” not compatible with
production DBA work
Incentives for Excellence
I was motivated by high expectations and corporate culture
Can also be naturally bright and curious Most need organizational incentive
State of Oracle Education
No critical thinking taught Emphasis on syntax and rote facts Which top Oracle gurus ever came out of OCP? OCP and non-OCP indistinguishable in
interviews
Senior review and mentoring
Underqualified DBAs often have no role models Pro-mentoring is cost-effective
Junior DBAs are half price Not primadonnas Smart junior DBAs become senior in just few years
Role of technology
Middle tiers and applications need instrumentation
Opportunities at: Design time Purchase time
Enables analytical methods
Conclusions
Problem is not empiricism vs. guessing Problem is sad state of excellence in DBA work Solutions:
High expectations Critical thinking skills Meaningful education (mentoring, NoCOUG) Instrumentation at the app layer Don't hire unqualified people