database administration for vista: lessons learned
DESCRIPTION
Database Administration for Vista: Lessons Learned. George Hernandez Amy Edwards Ezra Freelove University System of Georgia, Board of Regents July 11, 2007. Audience. Database Administrators Application Administrators Powerlink Developers. Who’s in the Audience?. Vista 3.x? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/1.jpg)
Database Administration for Vista: Lessons Learned
George Hernandez
Amy Edwards
Ezra Freelove
University System of Georgia, Board of Regents
July 11, 2007
![Page 2: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/2.jpg)
2
Audience
• Database Administrators
• Application Administrators
• Powerlink Developers
![Page 3: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/3.jpg)
3
Who’s in the Audience?
• Vista 3.x?
• Vista 4.x/CE6.x?
• BB Academic Suite?
![Page 4: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/4.jpg)
4
Panel Goals
• Share Perspectives
• Knowledge Exchange
• Stress Elemental View of Technology Stack
![Page 5: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/5.jpg)
5
Agenda
• What’s Vista?• Component Specifics• Database Performance• Backup and Recovery• Reporting• Capacity Planning• New Stuff• Reflections• Queries
![Page 6: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/6.jpg)
6
Our Team
• Ezra : <2yrs DBA experience, 10+ IT
• Amy: <4yrs DBA experience, 20+ IT
• George: 9+yrs DBA Experience, 12+ IT
![Page 7: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/7.jpg)
7
Our Deployment History
• Various individual and shared CE installs
• Beta’d 1.x
• Almost sank with 2.x
• Stabilized on 3.x
• 4.x appservicepack wha??
![Page 8: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/8.jpg)
8
Our Architecture
![Page 9: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/9.jpg)
9
What is a DBA?
• Database Management
• Application Installation/Administration
• Incident response
• Technology Cynic
![Page 10: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/10.jpg)
10
High Level Vista Opinions
• Incredibly complex
• Highly functional from a use perspective
• Tedious system practice
• Stated from the outset with a split personality
![Page 11: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/11.jpg)
11
Vista 2.x/3.x Split Schemas
AppSchema
(webct/vista)
ContentSchema
(ifssys)
- records - content
- logical object defs - content ACLs
- tracking/status - content rendering metadata
![Page 12: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/12.jpg)
12
Nasty Content
• Assessments/Surveys
• Discussions
• Grades
![Page 13: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/13.jpg)
13
Hierarchy and B+Trees
• Folderpaths are elemental to the system
• Rich content delivery requires expensive access methods
• CMSDK data structures meant to be accessed via Java– “wicked” nested records
![Page 14: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/14.jpg)
14
What is iFS/CMSDK?
• Obfuscate mime or raw content in the db
• Control Access via Java security level
• Deliver content via secure dbtier– Scalability of Oracle– Granular content controls
![Page 15: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/15.jpg)
15
Why I Think it Didn’t Make it to 4.x
• Licensure
• Complexity
• Bias
• Not portable to other databases
![Page 16: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/16.jpg)
16
Common Traits of the Vista Family
• Content stored in BLOBs
• Tracking
• Record Management for: – Hiearchy– Users– Security
![Page 17: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/17.jpg)
17
Performance Issues
• Typically originate from user actions on the apptier
• Long data retention means new sql explain plans
• Murky performance indicators
![Page 18: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/18.jpg)
18
Things to Check
• Statspack and archived performance metrics
• Normal maintenance
• Run suspect functionality in quiet systems and observe
• Profile of system or users
![Page 19: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/19.jpg)
19
How We Identify Database Issues
• Read and understand statspack reports • Examine specific metrics on
– cpu utilization– wait events– Latches– SQL execution
• Run captured queries manually• Provide data to support• Be Wary of First Assumptions
![Page 20: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/20.jpg)
20
Tools We Use
• SqlPlus
• Statspack
• Tora
• SqlDeveloper
• Nagios
• Bash,Excel
![Page 21: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/21.jpg)
21
Database Observations
• High transactional volume OLTP system • Oracle’s concurrency model handles most
issues• Schema design is mostly rational• High number of obscure Oracle components in
use– Java/XML in the database– BLOBs– Oracle Text
![Page 22: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/22.jpg)
22
Backups
• Impact to the online system/experience
• Timing
• Adequate sub-architecture
![Page 23: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/23.jpg)
23
Recovery
• Backup times versus recoverability
• How to handle logical Vista corruptions?
• Testing your recovery plan
![Page 24: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/24.jpg)
24
Vista Recovery
• Sincere admiration for deployments who have it enabled
• Expensive resource-wise
• Unique nuances of academic data
![Page 25: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/25.jpg)
25
Archives v. Backups
• Backups– Must be secure– Should be user-controlled
• Archives– Should be platform independent– Full application functionality not required– All instructional data spooled
![Page 26: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/26.jpg)
26
Reporting
• Active Sections
• Active Users
• Tracking Audits
• SIS synchronization
![Page 27: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/27.jpg)
27
Audits
• Session debugging
• Academic misconduct
• Research
![Page 28: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/28.jpg)
28
Ad Hoc Reporting
• Metrics grouped by role
• Content distribution*
• Template use metrics
• Issue Triage
![Page 29: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/29.jpg)
29
VPAR
• Attempt/Failure at warehousing tracking data and related schema objects
• Powersight data not deep enough
• 2.0 in early discussion
![Page 30: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/30.jpg)
30
Tools Used
• SchemaSpy
• Ifsbrowser/ifsshell
• HTML Db
• Oracle Warehouse Builder
• PL/SQL
![Page 31: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/31.jpg)
31
DB Capacity Planning
• Transaction Volume
• Redo Volume
• Trending previous growth
![Page 32: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/32.jpg)
32
What We Did
• Doubled DB resources every refresh
• We’ve reached the limit of monolithic servers
• More sizing work for RAC planning
![Page 33: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/33.jpg)
33
The Road Ahead
• Not Clear
• V4 migration elapsed times not feasible
• V3 aging somewhat gracefully but doesn’t have the longevity of CE
![Page 34: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/34.jpg)
34
Click What Our Last Major Fault Taught Us
• We can scale the apptier dynamically, but to what effect?
• No means to dynamically scale monolithic dbserver architectures
• Content store size isn’t our only liability
![Page 35: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/35.jpg)
35
Our Response
• Froze all Vista 4.x plans
• RAC Planning
• Virtualization Planning
• Assess full disaster recovery capabilities
![Page 36: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/36.jpg)
36
RAC
• 10gr2 only
• Oracle matured, efficient hardware plentiful
• Significant system administration changes
![Page 37: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/37.jpg)
37
Virtualization
• Logical extension from our large appfarms
• Node “build” process pre-existing
• Dynamic resource response via staged node images
• Assumption that Virtualization is the best means to utilized shared chassis hardware architectures
![Page 38: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/38.jpg)
38
Disaster Recovery
• Current– Physical standby on production scale
hardware– BCV splits of standby– Delayed tape spool
• Possible– Multiple standby targets with staged
virtualized node images– Two fully physical sites already in use
![Page 39: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/39.jpg)
39
Things to Read• Vista 2.x-4.x Powerlink dev guides• Powersight Kit Guide• 9.0.1-9.0.4 CMSDK javadoc• “The Data Warehouse Toolkit” by Richard Kimball• “Oracle SQL High-Performance Tuning” by Guy
Harrison• Oracle Technology forums• Bb wiki• Newsgroups• HiveMind/Tapesty framework docs
![Page 40: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/40.jpg)
40
That’s it, the rest is sql
![Page 41: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/41.jpg)
41
Queries• Full Tracking Auditset lines 220 pages 8000col user format a20col action format a30col pagename format a60col lcname format a50clear breaks computesbreak on User skip 1compute count of Action on Userselect tp.user_name "User",ta.name "Action",to_char(tua.event_time,'MM/DD/RR HH24:MI') "Time", tpg.name "PageName", lc.name "LCName", lc.id "LC_Id" from trk_person tp, trk_action ta, trk_user_action tua, trk_page tpg, learning_context lc where tp.id = tua.trk_person_id and ta.id = tua.trk_action_id and tua.trk_page_id = tpg.id and tua.trk_learning_context_id = lc.id and lc.id = '__' order by tp.user_name,tua.event_time/
![Page 42: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/42.jpg)
42
Queries• Active Vista User count
select 'ACTIVE='||a.active||chr(10)||'TOTAL='||b.logged_in
from
( select count(1) active from vista.wio_user_information where last_access_time>sysdate - 1/192) a,
( select count(1) logged_in from vista.wio_user_information) b/
![Page 43: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/43.jpg)
43
Queries• SIS Synchronization
select webct_id||','||remote_userid||','||sourcedid_source from person
where webct_id not like 'webct_demo%'
and learning_context_id = {learning_context.id of inst}
/
![Page 44: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/44.jpg)
44
Queries• Active Users
select lc.name||','||count(1)
from person p,learning_context lc
where lc.id=p.learning_context_id
and p.id in
(select distinct trk_person_id from trk_user_action
where event_time between '11-MAY-07' and '06-AUG-07’ )
group by lc.name
/
![Page 45: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/45.jpg)
45
Queries• Active Sections
select lc.name||','||lcs.name||','||lcs.source_id||','||count(1)
from learning_context lc, learning_context lcs, trk_user_action tua,
(select right_lc_id,left_lc_id from learning_context_index where left_lc_id in
(select id from learning_context where typelevel='20')) lci
where lc.id = lci.left_lc_id
and tua.trk_learning_context_id=lci.right_lc_id
and lcs.id = lci.right_lc_id
and lcs.typelevel='90'
and tua.event_time between '11-MAY-07' and '06-AUG-07'
group by lc.name,lcs.name,lcs.source_id having count(*) > 100
/
![Page 46: Database Administration for Vista: Lessons Learned](https://reader035.vdocuments.site/reader035/viewer/2022070401/56813691550346895d9e1c39/html5/thumbnails/46.jpg)
46
Want More?
• To view my resources and references for this presentation, visit
www.scholar.com• Simply click “Advanced Search” and
search by my username and tag: ‘bbworld07’
Sample Scholar Page
“Personalize and use this page if you’re using Scholar –
then delete this box.
Delete this whole page if you’re not using Scholar!”