information systems & computing leadership needed: creating and implementing an information...

Information Systems& Computing

Leadership Needed:Creating and Implementing an

Information Security Vision

M. Jost, D.Kassabian, D.MillarUniversity of Pennsylvania

EDUCAUSE 2004 Annual Meeting

Copyright Trustees of the University of Pennsylvania 2004. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given

that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.


About Penn

• The University of Pennsylvania was founded by Ben Franklin in 1740

• Penn is part of the Ivy League• Located in western Philadelphia• Community of more than 35,000 people


IT at Penn (1 of 2)

• Centrally provided administrative applications– Student, Financial and HR systems

• Distributed academic computing

• Centrally provided data network, PennNet:– 35,000 ports of 10/100 Ethernet– 200 campus buildings– 1.3 Gbps Internet access


IT at Penn (2 of 2)• IT funding, management, user support and

decision making are very distributed at Penn• IT governance and coordination is well established

– University CIO

– IT Roundtable - Computing Directors

– SUG - IT support professionals and developers

– NPTF - Financial planners, networking charges

– NPC - cross section, developing network policies


Situation in Summer’03• Security administration, including patching

of operating systems, uneven across campus• Anti-virus software on most computers in

campus departments, but fewer in residences• Student systems becoming compromised at

an alarming rate• Staff time costs in responding to security

compromises was skyrocketing


Estimated cost of Blaster/WelchiaFor August 2003

ITEM EST. COST

1200 compromised machines

-Manage detection and notification

-Format and rebuild machines

-Remove Blaster from machines

15-25%

15-25%

15-25%

9,000 Vulnerable machines (patched twice)

-20 campus-wide scans, 14 mass notifications of

vulnerability

-4,500 Patch automatically (twice)

-4,500 Patch manually (twice)

2-3%

1-2%

30-40%

Total $287,000

Lost productivity of faculty/staff machines disconnected ?


Why not rely on perimeter firewalls?Campus firewall not a panacea

University Date Netbios ports blocked

# Windows machines

# infected

% infected

Penn 9/11/2003 11,000 1,200 11%

Large state university 7/28/2003 12,000 1,500 13%

Ivy League peer 1/2/2002 18,000 3,146 17%


Freedom and Responsibility in personal computing

• Can we mandate desktop security practices? Do we reduce user freedom in in the name of security?

• Many in academia are accustomed to great freedom in managing their computers and software

• Freedom and autonomy are not necessarily at odds with security and legal risks– (with freedom comes responsibility)

• We needed security vision, complete with user education, balanced policy, technical support, and adequate funding to improve security


Security Vision and PlanVision• Get the user community on board• Make security the default wherever possible• Promote achievable, affordable plans• Treat security vision as an ongoing process with a two-year horizon

Plan• Develop defense-in-depth layered technical approach, including:

– Prevention: Anti-virus, patching, secure configs, limited filtering / firewalling– Detection: Vulnerability scanning, IDS, server log reviews– Response: removal of compromised hosts from networks, limited network

filtering, strong communications• Develop policy, best practices, and end-user communication• Communicate benefits to users and their IT support professionals• Secure sustainable funding for the key initiatives


Building Campus Consensus• Risk Assessment to establish priorities

– Security discussed informally among security, networking, other central and distributed IT organizations

– General consensus on the layered approach• Prevention: Patch management, anti-virus, education• Detection: intrusion detection, vulnerability scanning• Response: locating machines and incident management


Building Campus Consensus• Network Planning Taskforce

– Security discussions over several meetings– Risk discussed in terms of prioritization and specific components

of the layered approach– Specific components discussed

• Patching– Managed vs Unmanaged– Options: Windows, Software Update Server (SUS), Systems

management server (SMS), commercial products e.g. HFNetChk Pro, PatchLink, BigFix

– Special challenges patching students» Machines we don’t own but need participation from owners» Privacy issues



– Specific components discussed• Virus filtering on mail servers

– Campus-wide vs individual servers

• Firewalls/routing, VPNS, personal firewalls– Explained the concepts, terminology and how each work– Discussed pros and cons of different types of implementations

• Secure out of the box– Default images with strong authentication on truckload sale and

Penn machines



– Specific components discussed• Vulnerability scanning

– Past results– Options to implement at a local level

» Develop tools for local support providers– Options to implement on the network at a central level

» IDS boxes, router flow logs

• Better ways to locate compromised and vulnerable machines

– General agreement on security direction for Penn reached


Building Campus Consensus• Network Policy Committee

– Worked to establish the policy needed to support the agreed to direction

– Earlier had implemented standards for the most critical machines on campus

– Recently approved: PennNet Computer Security Policy (Patch Management Policy)

• Critical updates to all campus connected systems must be applied within three business days or computer may be disconnected from the network

• Approved in June 2004; Implemented in September 2004


Building Campus Consensus• Patch Management Policy

– Draft out to IT community for review before final approval

– Comments and changes strengthened the policy– Discussed with University management to gain

appropriate support as policy has broad impact– Communicated to the campus community through

several types of communications• University publications, newpaper, newsletter• Presentations to several IT groups throughout campus


Building Campus Consensus• Summary

– Many people were involved– Discussion at different levels within the organization– Education of community

• Prioritization using a layered approach• What technology was available and how it worked• Options and costs to consider

– Input welcomed and incorporated into the solution– Participation from community resulted in best solution– Implementation plan developed


Funding• Funding the Implementation Plan

– Costs of implementation were estimated• Included costs across all of the central IT organizations, not

just networking and security

– Funding Source options considered– Start with the most likely

• Central University Funds• NPTF – group that helps set annual user fees for network


Funding

• Funding Options – University funding for central organizations severely

constrained

– NPTF liked the plan, hated the cost• Funding for the schools constrained

• Looking for network costs to remain flat or go down

• Wanted a more secure network without additional costs

• Believed central university funding should pay for security or students should pay for their share of the burden


Funding

• Students– Student behavior and computing support structure was

a large cost driver

– Bring unpatched, sometimes infected machines back to campus and plug into network

– Support for undergraduates in residences provided by student residents not University employees

– Limited or no support for Fraternities and Graduate students on campus, off-campus students bringing laptops on campus


Funding

• Funding Proposal– Identified the portion of costs attributable to faculty and

staff and separated it from costs attributable to students– Identified key executives financially responsible for

student support – Developed presentation to educate execs about the need

for security and the cost of delivering it• Target audience was Business Administrators, not IT personnel• Framed in terms of productivity loss of both end users and IT

support personnel due to Blaster


Funding

• Meetings to Look for Funding– Met individually with several key executives that dealt

with students• Agreed that money was needed and issues were valid

• No initial agreement on where it should come from

• Agreed to support a plan for funding to come from student fees via the organizations who collected the fees

– Final meeting with all the key players • Agreed on student funding for ongoing costs


Funding• Final Funding Sources Identified

– Network Charge would include funding for faculty and staff

– Student resident fee (not rent) would increase to cover undergraduates on campus

– Fraternities would pay a surcharge for their network connections

– Graduates and off campus funds would come from the Provost and/or central University funding

– One time costs to implement would be paid for by central IT organization


Funding• Summary

– Funding constraints made it impossible to receive all required funding from existing funding sources

– Tension between responsibility for funding students and funding faculty/staff played an important role in final solution

– Case for additional funding requests needed to show the benefit/added value the plan would deliver to those paying for it

– Educating customers on those benefits is a critical success factor


Implementation• Challenges: 500+ LSPs, 30,000+ end users• Leverage points:

– PennConnect CD (Internet Connection Firewall)– Back-to-School Truckload Sale– Prizes and drawings to build awareness– Mass email, banner ads in Daily Pennsylvanian– Vulnerability scanning– Supporting patch management service


Implementation• Tasks

– Evaluate firewalls

– Communications plan

– Secure Out of the Box – (Dell & IBM images)

– PennConnect CD

– Security awareness quiz (iPod giveaway)

– Implement patch management service & supporting documentation

– Contingency plans for router filtering

• Phased implementation:• 8/04 – 9/04 - Communications and awareness

• 9/04 – 12/04 – Vulnerability scanning and “warning letters”

• 1/1/05 – Disconnect machines not in compliance


Implementation

• Communications Plan– Identify target audiences (students vs.

faculty/staff vs. LSPs)– Identify key messages (“enroll in patch

management” vs. establish a patch management service for your users”)

– Develop a “media plan” – target vehicles, dates, deadlines, etc.


Media Plan

Month Event name13-Jul Policy article appears in AlmanacJul - last weekLaw School - LLM Program -

Registration/orientation/classes begin26-Jul PennConnect CD - release 1

Aug - 1st MondayWharton - MBA Orientation & pre-term events begin11-Aug Med School - 1st year orientation & classes beginAug - Mid-monthDental School - Freshman Orientation 16-Aug Almanac September Events calendar deadline


Implementation


Results • We now have a program to keep systems secure,

rather than dealing with everything as a “one-off”• 5300 students/1000 faculty staff took the security

quiz• 72% fewer machines compromised Fall, 2004 vs.

Fall, 2003• Overall sense of campus IT leadership that Fall,

2004 went a lot more smoothly than 2003, though we were also lucky


Lessons Learned• Present security initiatives as a business case. Measure the

cost of poor security whenever you can. ROI’s sell projects.

• Gain support with appropriate discussions from operational management to executive level.

• Make strategic planning and budgeting processes transparent to your clients. It was truly amazing to be able to establish a campus consensus for mandatory security standards for all campus machines.

• Structured and consultative policy development, with a thorough vetting process, yields workable, enforceable policies with a high probability of changing behavior.


Lessons Learned• Don’t try to go too fast. Allow time to assimilate

change.• Funding models should drive costs back to their

source; administrative units don’t like footing the bill for residential student security problems.

• Always coordinate end user communications with LSPs.

• Support large policy changes with a robust, targeted communications plan, supporting services and documentation.


Lessons Learned

• Look for leverage points in developing strategies (patch management) and implementation plans (communications, incentives, mass communications).

information systems & computing leadership needed: creating and implementing an information...

Documents