www.ts.vcu.edu

How We Maintain a Healthy Bb Environment

Sheila Chandler

Director, Learning Systems

Virginia Commonwealth University

Presentation Overview

• Background– Who am I?– Virginia Commonwealth University (VCU)– VCU Blackboard Environment

____________________________________

How We Maintain a Healthy Bb Environment

Sharing Lessons Learned

Who am I?• High School – Computer programming (IBM)• University of Louisville

– Sociology Degree– First programming job (PDP 11/45 - BASIC)

• Community Hospital of Central California (MUMPS, BASIC)

• Virginia Commonwealth University (VCU) -Accounting Degree

• Auditor of Public Accounts – CPA, CISA• VCU

– Programmed on Web Course in a Box (1998 to 2000)– Blackboard purchased WCB (2000)

About VCU

• Richmond, VA

• Students: 32,000+

• Faculty: 1,927

• Adjunct: 1,161

• Staff: 2,586

• Monroe Park, MCV, Qatar, NOVA

Bb App Environment• Bb Application (2000 to present)

– Bb Classic Enterprise (7.3 -> v9 – May 2010)– Bb Content Management System

• Building Blocks– Bb Faculty Tools (VCU developed)– CPS Connection (Clickers)– Echo360 Content Creator – Health Check (modified)– Help Desk Tools (VCU developed)– Wimba– SafeAssign– SoftChalk

Courses per Semester

0

500

1000

1500

2000

2500

3000

3500

# of Courses 656 612 956 1244 1447 1726 2070 2079 2569 2337 2777 2612 2987 2993 3170

Fall 2001(CourseInfo4)

Spring 2002 (Bb

5)

Fall 2002 (5)

Spring 2003 (5)

Fall 2003 (5.5)

Spring 2004 (5.5)

Fall 2004 (6)

Spring 2005 (6)

Fall 2005 (6.3)

Spring 2006 (6.3)

Fall 2006 (6.3)

Spring 2007 (6.3)

Fall 2007 (7.1)

Spring 2008 (7.1)

Fall 2008 (7.3)

Bb User Environment

Fall 2008

3319 Courses

2031 Instructors

29688 Students (85%)

723 Organizations

Spring 2009

3099 Courses

1983 Instructors

28236 Students (87%)

804 Organizations

Lesson Learned #1Designate Bb Coordinator and

Support Team

• VCU Support Team (Tribe)

– Technology Services• Learning Systems• Help Desk• University Computer Center• Networking

– Center for Teaching Excellence– Support staff in Schools (School of Nursing, Business, etc)– Vendor (Blackboard)

Lesson Learned # 2Recognize Mission Critical

• Path to Mission Critical– Student demand (85%+)– Faculty use (email, gradebook,

document storage)– Administration/Staff found uses– Keep statistics – show growth

Lesson Learned # 3 - Scale Out

Lesson Learned #4 Request System Audit

Lesson Learned # 5 Monitor and Take Action

• Hobbit (Internal)

• Pingdom (External)

• Custom Scripts– Check db connections– Search logs for Oracle errors– Scan Snapshot files (header & footer)– Monitor SAN space (set to expand at 90%)

#5 Monitor and Take Action

• System cleanup– Rotate logs– Run Bb Cleanbb once a week– Restart app servers twice a week

• Track usage and react to trends• Ask Users

– My VCU Portal Polls– Annual Tech Fair suggestion box– Talk to Students on Campus– blackboard@vcu.edu

#5 Monitor and Take Action

• External Listserv– ASU list

• (BBADMIN-L@lists.asu.edu)

– BIE list • (BIEACAMDEMIC@lists.blackboard.com)

– Developers • (BB-OPENSRC@lists.vcu.edu)

• Internal VCU Listserv– Bb-core– Bb-mgmt– Bb faculty list

Lesson Learned #6 Document and Communicate

• Technical– Bb Diary– Internal wiki – notes– BbCMS – Documentation– Emergency Call list

• End-User– Help Desk - blackboard@vcu.edu– Blogs (Tech Services, Bb system-wide,

Bb faculty)– Learning Systems Newsletter– Technology Services Knowledge Base

(http://www.ts.vcu.edu/kb/blackboard.html)

#6 Document/Communicate

• Regular Meetings– Weekly LS meeting– Monthly Bb Tech meeting (agenda & minutes)– Monthly Center for Teaching Excellence

meeting– Periodic meetings with IT Directors of Schools

#6 Document/Communicate

• Special Meetings– Special Task Force (iptables fix)

• Discovered that net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 1 doesn't survive an iptables restart

– Bb TSM bi-weekly meetings• Periodically request to review Bb tickets

Lesson Learned #7“No such thing as a SMALL change”

• ALL Changes follow Written Procedures– Development Environment (maybe multiple

times)– Test Environment (always multiple times)– Production

#7“No such thing as a SMALL change”

• System Maintenance – Regularly Scheduled– Second and Fourth Friday of the month– Perform Downtime Checklist

• One week before downtime• During downtime• After downtime

– Group IM (Instant Messenger)– Follow emergency communication plan (if needed)

Lessons Learned Summary

1. Designate Bb Coordinator and Support Team

2. Recognize Mission Critical

3. Scale out

4. Request System Audit

5. Monitor and Take Action

6. Document and Communicate

7. “No Such Thing as a SMALL Change”

Additional Information

• Additional Information– Sheila Chandler (sdchandl@vcu.edu)– Bb Connections site– http://healthybb.wetpaint.com

• Thank you to…– Learning Systems Staff– University Computer Center Staff– Networking Services– Center for Teaching Excellence