county of sacramento gtc west 2007 information security

COUNTY OF SACRAMENTO

Helping you get the most out of what you put into protection information

County of Sacramento Office of Communication and

Information Technology 799 G Street

Sacramento, CA

GTC West 2007 Information Security – Partners in Learning

CASE STUDY

NETWORK VULNERABILITY ASSESSMENT AND MITIGATION

A Real Government Dilemma – Sacramento County Project Implementation and Lessons Learned

Network Vulnerability Assessment and Mitigation June 21, 2007

County of Sacramento

Board of Supervisors

2007

Roger Dickinson, 1st District

Jimmy Yee, 2nd District

Susan Peters, 3rd District

Roberta MacGlashan, 4th District

Don Nottoli, 5th District

Terry Schutten

County Executive

www.SacCounty.net

Published by:

Pat Groff, Chief Information Officer

Office of Communications and Information Technology


799 G Street

Sacramento, CA 95814

(916) 874-7825

May 2007

Page ii

June 21, 2007 Network Vulnerability Assessment and Mitigation

TABLE OF CONTENTS 1. OVERVIEW................................................................................................................................................7

1.1 EXECUTIVE SUMMARY.........................................................................................................................7 1.2 THE OUTLINE .......................................................................................................................................7

2. THE ‘WHO, WHAT, WHY’ OF THIS.....................................................................................................9

3. SCOPE OF DISCUSSION .......................................................................................................................10 3.1 NINE QUESTIONS PARTNERS IN LEARNING ATTENDEES ASKED BEFORE HAND:...................................10 3.2 ENTERPRISES MUST PREVENT AND LIMIT DAMAGE TO ITS BUSINESS OPERATIONS ..............................10 3.3 ELECTRONIC DATA IS UNDER ATTACK..............................................................................................11 3.4 NEW SECURITY STRATEGIES ARE NEEDED.........................................................................................12 3.5 INFORMATION PRIVACY AND SECURITY ROUNDTABLE......................................................................13 3.6 VULNERABILITY MANAGEMENT ........................................................................................................14 3.7 SACRAMENTO COUNTY MOVES TOWARDS A COUNTYWIDE IT PLAN.................................................15 3.8 THEMES EMERGED FROM THE IT PLANNING FOCUS GROUPS.............................................................16 3.9 FOUR REASONS WHY WE DEVELOPED A COUNTY-LEVEL IT PLAN ..................................................17 3.10 A MODEL OF THE SECURITY PROGRAM..............................................................................................18

4. TECHNOLOGY SPACE .........................................................................................................................20 4.1 PROTECTING NETWORKS, SYSTEMS, DATA, AND APPLICATIONS FROM THREATS ..............................20 4.2 NETWORK VULNERABILITY ASSESSMENT (VA) CAPABILITY ............................................................21 4.3 WHAT IS VULNERABILITY? ................................................................................................................22

5. CASE STUDY: VULNERABILITY ASSESSMENT ............................................................................23 5.1 PROPER PLANNING.............................................................................................................................24

5.1.1 About Us .......................................................................................................................................25 5.1.2 A little more about us… ................................................................................................................25 5.1.3 Starting to solve the problem........................................................................................................26 5.1.4 Planning Continues with Knowing What You Need......................................................................26 5.1.5 Requirements of a solution ...........................................................................................................26 5.1.6 The Network Vulnerability Assessment (NVA) Vendor landscape: ..............................................27

5.2 CAREFUL DEPLOYMENT.....................................................................................................................28 5.2.1 Deployment Consideration and Implementation Tips ..................................................................28 5.2.2 Our Deployment ...........................................................................................................................29 5.2.3 Deployment: Specific Considerations...........................................................................................30

5.3 DETAILED FOLLOW THROUGH ...........................................................................................................33 6. UNIVERSAL LESSONS LEARNED......................................................................................................34

6.1 VULNERABILITY ASSESSMENT ACTIVITIES ........................................................................................34 6.1.1 Multiple forms of scanning ...........................................................................................................34 6.1.2 Deploy correctly ...........................................................................................................................35 6.1.3 Scan frequently and target new vulnerabilities ............................................................................35 6.1.4 Vulnerability Management lifecycle functions..............................................................................35 6.1.5 Support compliance needs ............................................................................................................35 6.1.6 Don’t use Vulnerability Assessment as a management tool..........................................................36

6.2 TEN RECOMMENDATIONS TO LIMIT THE PROBABILITY OF A SUCCESSFUL ATTACK ...........................36 6.3 CONSIDERATIONS AS WE MOVE FURTHER ALONG WITH IMPLEMENTATION.....................................37 6.4 COSTS ................................................................................................................................................38 6.5 FOR US, IT IS TIME FOR ACTION! .......................................................................................................38 6.6 THE NEXT STRATEGIC ISSUES FOR US…............................................................................................39 6.7 TIMELINE, PREREQUISITES AND CO-REQUISITES OF SECURITY FROM THE BUSINESS SIDE.................40

7. CONCLUSION .........................................................................................................................................41

Page iii


7.1 THE CRISIS ........................................................................................................................................ 41 7.2 VULNERABILITY ASSESSMENT .......................................................................................................... 41

Page iv


A good security programgoes unnoticed

But…

A bad security program, on the other hand, has the power to

ruin all your efforts

A Case Study

Network Vulnerability Assessment and

Mitigation

GTC WEST 2007 Partners in Learning Edition

Developed by Jim Reiner, Chas Lesley, and Michael Walters

A real government dilemma – Sacramento County Project Implementation and Lessons Learned

Page v


1. OVERVIEW

1.1 Executive Summary Many organizations are under pressure to improve their security posture. That’s no surprise to many of us who see the need to secure our information from outside threats as well as from internal vulnerabilities.

Securing your organization is filled with business and technical challenges, endless assaults on your network, and a tangle of products and services that can make it harder instead of easier to secure information.

This problem of securing the network is a serious issue for organizations trying to make themselves more transparent, their information easier to access, yet secured from unauthorized use. Not only can it put an organization’s strategy at risk, but it can also degrade productivity, expose sensitive information, and undermine customer confidence.

A growing body of research explains the dilemma, and lays out recommendations to reduce the risk. In light of this, Sacramento County is improving its vulnerability management capability. The place to start in the vulnerability management process is with a network vulnerability assessment capability.

Proper planning, careful deployment, and a detailed follow-through on your action plan are critical to a successful deployment of vulnerability assessment technology. By taking these three steps, you will be well on your way to improving your security posture.

This case study shares our experience from a “before, during, and after” perspective.

1.2 The Outline The outline of our case study

1. The ‘who, what, why’ of this

2. Scope of discussion

3. Technology space

4. Case study: Vulnerability Assessment

♦ Proper Planning

Overview


♦ Careful deployment

♦ Detailed follow through

5. Universal lessons learned

Overview


2. THE ‘WHO, WHAT, WHY’ OF THIS Why are we here today?

• To share information.

• We’ll present what we learned during a multi-year effort.

And why are you here today? (or why are you reading this?)

• Save yourself time and effort

• Gain insight in this technology space

• Better understand some of the processes and policies needed

Why is this, the topic for the GTC Partners in Learning?

Those planning this event conducted preliminary surveys to determine topics of interest. Respondents indicated an interest in wireless deployment, network standards, combating hackers, and vulnerability assessments. As we whittled it down, the network vulnerability assessment topic rose to the top. All we needed was a local government organization willing to share what they are doing.

How does vulnerability assessment fit into all the things you can be working on to secure your assets? We’ll take a quick look at the big picture to see why Sacramento County has made this its current priority. And we’ll conclude by telling you about the next serious issues we are already looking at.

This is a case study about deploying network vulnerability assessment technology in Sacramento County. This is a tool to scan endpoints in the network for vulnerabilities. The effort is a work in progress. We are still learning and implementing this ourselves. However, there are some key lessons learned that we can share. Really, these are lessons that have universal application in any of your security initiatives.

This booklet includes additional information we think you will find helpful. Not all of this will be, or can be, presented at the live Partners in Learning session.

The ‘Who, What, Why’ of This


3. SCOPE OF DISCUSSION We’re all asking questions about vulnerability assessments

3.1 Nine questions Partners in Learning attendees asked before hand:

1. When we find vulnerabilities, how do we get management to understand the importance and the need to fix them as soon as possible?

2. How do I procure a trusted company to do the vulnerability testing?

3. What should I ask for in my request for proposals sent to vendors?

4. Do I want vulnerability testing, penetration testing, development of a security plan, etc.?

5. What should our department’s technical staff be doing to test for vulnerabilities? Is it only related to the network or is there more (like physical or operation testing, pc testing, policy review, etc.)

6. What exactly is vulnerability testing? It seems like there are many approaches and ways to go about it.

7. How do I write a statement of work so that it is concise and to the point, so the vendor will know what I want done?

8. Are there products out there that we should consider procuring that will allow us to do continuous vulnerability testing? How much do they cost? How much work is involved in setting them up and monitoring them? How many staff does it take to monitor it?

9. If we bring in a contractor, how often should the vulnerability or penetration testing be done?

The rest of this document will answer these as we describe our effort.

3.2 Enterprises must prevent and limit damage to its business operations

But, it is not possible for most organizations to prevent all bad things from occurring. The goal of a security program should be to limit the probability of a

Scope of Discussion


successful attack and, if an incident does occur, limit its impact on the organization.

Enterprises must prevent and limit damage to their business operations by deploying policies, processes, and technologies to detect and block attacks – both internal and external – and minimize the vulnerabilities that enable attacks.

At the same time, the enterprise threat environment is changing rapidly. Threat and vulnerability management strategies must change with them.

Does this characterize your organization:

♦ Your IT is not consistent throughout.

♦ Everyone does what they think is best.

♦ You have vulnerabilities throughout your organization.

♦ Redundant services and attendant costs result from this inconsistency, as well as duplication of efforts.

♦ These inconsistencies affect customers and key players.

Then, let’s find out how we can protect enterprise networks, data, applications, systems, and IT infrastructure from external and internal threats.

3.3 Electronic Data Is Under Attack

Security professionals are under pressure to improve their organization’s security posture. This is driven by a hostile-threat environment and compliance requirements. It is also complicated often by not having an organization-wide security understanding.

If an organization does not understand its vulnerabilities and exposures, it is not possible to properly secure the environment.

Most organizations focus security resources on reactive security technology that offers value at the time of the incident, or after it occurs. This is important; however, maturing your security program requires implementation of pre-incident, proactive technology controls and processes.

We learned some lessons from scanning for “I love you” & “slammer.” We knew who was vulnerable and then sent a weekly report to departments and we got a reply back. We did this for 6 months.

Scope of Discussion


But, we were never able to clean up 100% of the hosts. The departments were too busy and couldn’t get to it, or they couldn’t get the PC off the network, or their applications were not patchable.

There were lots of reasons for exceptions. Risk level came down for a while then went back up in spikes as the County introduced more nodes without patching known problems which in turn reintroduced the worms.

3.4 New Security Strategies are Needed Privacy and security breaches threaten our stability.

Ever changing threats, technologies, and business practices demand new security strategies.

IT security and operations organizations have developed defenses and processes to deal with mass attacks – for example worm attacks of recent years – but financially motivated cybercrime and other targeted attacks are now having a far greater impact on business.

Attacks now are often higher in the stack – attacking apps such as the web browser or lower in the stack – installing rootkits or corrupting drivers… more subtle and less noticeable.

This threat shift which will certainly continue is driving the need for improved data protection and activity monitoring.

Consider the changing nature of vulnerabilities as you prepare your tactical and strategic security initiatives for the future. Examples where an organization is limited in being able to find and fix vulnerabilities before they are exploited: mobile devices with embedded commercial OS, app svc providers, and appliances with stripped down operating systems.

High level issues that we want answers to:

1. Which new technologies and attack strategies will expose your IT systems and data to security breaches?

2. What are the most effective technologies and best practices to protect networks, systems, applications, and data?

3. What are the most effective technologies and best practices for managing and auditing IT configurations and policies to eliminate vulnerabilities?

4. What are the leading providers of security infrastructure products and services and what are their key offerings, differentiators, and strategies?

Scope of Discussion


3.5 Information Privacy and Security Roundtable March 2007 with County Management

Forces are driving us to take action, as well as holding us back.

You are concerned about the security of your information. What is driving you to take action?

• Minimize liability

• Maintain customer confidence

• Use best practice

• Public accountability

• Public expectation of transparency

• The push for expanding e-government

• Accessibility with sufficient safeguards

• Sensitivity to ID theft notification

What is restraining, or holding you back, from taking action?

• Selling it – “buy in” at the top

• Time sensitivity – “I don’t have time for this”

• Technological complexities

• Expense – this stuff costs

• Constraints of uniform standards

• Business processes are inconsistent

• Legal requirements

• Difficulties in focusing on “what it is & what to do”

• It’s overwhelming to us

• Competing priorities

Scope of Discussion


This is a gold mine of info!

Why? Because management told me exactly what their ‘hot buttons’ are and what their ‘pain points’ are.

Will I use this information? Absolutely! And so should you!

3.6 Vulnerability Management You need to be anchored with a well grounded vulnerability management program.

Vulnerability Management (VM) increases the effectiveness and efficiency of enterprise security.

The VM lifecycle offers guidance on operational processes and technologies needed to find and remediate security weaknesses before they are exploited.

Without a complete and correlated view of various assessment technologies it is difficult for an organization to really understand its security posture.

Examples of assessment technologies that specialize in specific elements of the environment: web app scanners, security source code scanners, wireless scanners, database security assessment scanners, automated penetration testing tools, VoIP security scanners, and network behavior analysis, security and event management tools.

Vulnerability Management lifecycle common functions:

1. – definition of policies that define a secure infrastructure

2. – baseline to discover vulnerabilities and security configuration policy compliance issues

3. – monitor the environment for security events that require some action on the part of IT security or operations

4. – improve security via mitigation activity as a result of baseline and monitoring functions

5. – risk assessment to prioritize mitigation activity

IT organizations that are evaluating or implementing VA should evaluate product support for VM lifecycle functions needed to assess risk, support mitigation activities, and support compliance reporting.

Scope of Discussion


3.7 Sacramento County Moves towards a Countywide IT Plan

You will have better success with vulnerability management if you have three things in place:

1. IT Strategic Plan that includes security as a key focus area

2. IT governance that formalizes processes to make decisions and manage IT

3. A security program, staff, and a budget that can take this on and coordinate the efforts

Prior to 2003, the County’s IT infrastructure had migrated to a largely decentralized environment that was not based on a clear Countywide IT strategic plan. Decisions on technology and support requirements were often made without consideration of the infrastructure as a whole (and still are to some extent.)

The decentralization in the County’s IT environment, driven by a lack of customer satisfaction with the services provided by the previous central support department, led to fractured resources and reduced centralized support. Agencies and departments found ways to address their IT requirements without seeking the assistance, advice, or reliance upon the central IT department.

As a consequence, IT solutions were developed independently without factoring in long-term service requirements, economies of scale or infrastructure standards. This lack of overall strategic planning created barriers to integrating departmental systems and increased the cost to maintain multiple systems.

Technology decisions were also often made without an exploration of the underlying business issues driving the perceived technology needs. The County historically had problems identifying the benefits of IT, the costs of development and operations, and the impact and value of applications. Systems built in the past have cost more than expected and have also taken longer than planned to build. During the 1990s, the departments highlighted these problems and the need for corrective action.

Sacramento County was not alone in the challenges it faced. In the February 2002 issue of the magazine Governing, a special report graded the top 40 U.S. counties on management issues including information technology. The report indicated that most senior information officers think it is an excellent idea to do long-term planning.

So, in 2003 we developed and adopted a countywide IT Plan.

Scope of Discussion


3.8 Themes Emerged from the IT Planning Focus Groups The Sacramento County IT Plan includes security as a priority item.

Participants in focus groups said they want to:

1. Manage IT with a Countywide perspective

2. Improve service delivery

3. Expand access to information and services through electronic media

4. Enable and improve communications

The focus groups identified ten business drivers – external and internal forces that drive customer expectations and satisfaction, executive expectations and directions, best practices, and reasonable costs to taxpayers.

These business drivers include:

1. Federal, state, and local laws and regulations

2. Public health and safety of our community

3. Public’s demand to access information and services

4. Effective government services to the region’s citizens

5. Timely, accurate, and responsive communication to constituents and employees

6. Business strategies implemented with a Countywide perspective

7. Privacy and security

8. A highly skilled and well-trained County workforce

9. Information access and sharing between departments

10.Limited financial and human resources

Sacramento County IT Plan and Process:

http://www.saccounty.net/itpb/it-plan/index.html

Sacramento County IT Governance:

http://www.saccounty.net/itpb/standards-policies/it-constitution/index.html

Scope of Discussion


3.9 Four Reasons Why We Developed A County-Level IT Plan

Business needs drive our security goals and objectives of the IT Plan.

1. Support the business needs

2. Provide an enabling infrastructure

3. Provide an IT roadmap that is consistent with the County Strategic Plan

4. Gain insight from various business groups about their priorities, services, programs and plans

We needed to take a holistic, global view that unified the technology architecture with a plan that is governed across the enterprise.

The IT plan is a portfolio to move incrementally to fill gaps in our program and service delivery – consistent with and in concert with the County strategic plan. The plan helps us manage IT demand collaboratively with consistent service and value while focused on business drivers.

One of our three focus areas is the infrastructure

Enhance the County IT Infrastructure to provide a robust, stable, scaleable and secure foundation

The three year goals established in 2003 included:

♦ Improve security for people, buildings, and data

2005 Objectives included:

♦ Implement a vulnerability assessment solution for all County network-attached devices.

One theme that was repeated by virtually every contributing department is that information technology and security is no longer an “added” component to our processes, but rather an integral part of how the County does business.

To build a secure environment, you must have order.

TIP: Advancing security initiatives is a little easier if your organization has a business plan or IT Plan that acknowledges the importance of security. You’ll need the ‘buy-in.’

Scope of Discussion


3.10 A Model of the Security Program Vulnerability management is part of our information security program.

Here is a model of the security program used in our central IT organization

Governance

Security Professionals

Employee Training

Security Monitoring & Auditing Controls

Network vulnerability assessment is one of the controls used to evaluate the security of the environment. We’re using a holistic approach for managing information security based on ISO17799. A detailed description of this security program as well as templates is available on the CCISDA website at www.ccisda.org in the document library.

The necessary components for managing and controlling assets include the following.

1. Information Security Officer – develop and enforce the security program

2. Security Committee – division representatives review and update the information security program and policies as necessary

3. Information security policies – high-level guidance that directs the organization

4. Security awareness & training – recognize / understand responsibility and accountability

5. Information identification and classification – standards by which information resources are managed and accessed

6. Information risk assessment – formal analysis process to determine reasonable and appropriate security controls for business

Policy and Procedures

Business Continuity & Disaster Planning

Information Classification

Information Risk Management

Scope of Discussion


7. Implement information security controls – protecting information assets through administrative, technical, and physical safeguards

8. Monitor effectiveness and assurance - determine compliance with security standards

9. Business continuity and disaster planning – preservation of business in the face of major disruptions

Scope of Discussion


4. TECHNOLOGY SPACE Vulnerability Management is just one type of security control we need.

4.1 Protecting Networks, Systems, Data, and Applications from Threats

Network security – protection of systems and applications and data begins with good perimeter and network security, which focuses on strategies and technologies to protect enterprises’ network and IT infrastructure from external and internal attacks.

Network access control – the worm attacks of recent years created demand for a NAC process that would prevent corrupted and dangerous systems from gaining network access. NAC efforts improve defenses against corrupt and dangerous systems by interrogating a node as soon as it is plugged in.

Vulnerability management – VM focuses on operational processes and technologies needed to discover and remediate security weaknesses before they are exploited. Technologies include VA, security configuration mgmt, patch mgmt, and security event management.

Data protection – focusing on technologies and strategies to protect information where it is stored and as it is used. High-profile data loss events continue to occur with significant financial fraud reported.

Secure messaging and web content – content inspection, compliance, and retention policies cut across all media. Email is one of the most important and visible enterprise security pain points because of threats such as spam, viruses, spyware, and phishing. Regulations are driving the requirements for improved outbound scanning and encryption.

Application protection – Financially motivated attacks are increasing at the application level as more applications are exposed directly to the Web internally or externally. Applications frequently fail because they have not been built with security in mind.

Mobile security – the majority of mobile and remote user devices, including home PCs, notebooks, PDAs, and smartphones, are not adequately protected. User-owned equipment and mass storage capabilities have only made the situation worse. Many of the largest data breaches have involved the loss of mobile devices containing customer data.

Endpoint security – desktop antivirus products are the largest enterprise security market segment. Growing ineffective antivirus technologies are being augmented by proactive host-based intrusion prevention systems.

Technology Space


Virtualization and virtualized security – this offers opportunities to reduce costs and increase agility and new ways to package security. But it also presents new security threats. Traditional ways of securing physical servers aren’t adequate for virtual machines.

4.2 Network Vulnerability Assessment (VA) Capability

The first step in the vulnerability management process is a network vulnerability assessment (VA) capability

VA technologies attempt to limit their impact on the environment through the use of non-intrusive scanning. VA tools focus on the state of the endpoints and provide report data that can be used to improve your organization’s security posture.

And here again are those nine questions Partners in Learning attendees asked before hand:

1. How do we get management to understand the importance of fixing vulnerabilities as soon as possible?



4. Do I want vulnerability testing, penetration testing, a security plan, etc.?

5. What should our department’s technical staff be doing to test for vulnerabilities?

6. What exactly is vulnerability testing?

7. How do I write the functional requirements so the vendor will know what I want done?

8. What products are out there for vulnerability testing? How much do they cost? How much work is involved in setting them up and monitoring them? How many staff does it take to monitor it?


Technology Space


4.3 What is Vulnerability? It is a condition or weakness within a system that could be exploited to produce harm or loss of data.

Examples:

♦ PC’s not at current Service Pack level

♦ Missing OS patches on Servers

♦ No passwords set on common accounts

♦ Misconfigurations

Technology Space


5. CASE STUDY: VULNERABILITY ASSESSMENT Perspectives from the County of Sacramento

How to deploy vulnerability assessment successfully

proper careful detailed planning deployment follow through

Step 3Step 2Step 1

VA is an important part of effective security program and forms the basis of the VM process.

VA provides foundational discovery and security baseline data as one of the first steps in the VM process and it provides ongoing data through periodic re-scanning.

VA tools provide a bottom up security baseline of an IT environment with respect to a database of known vulnerabilities.

The value of vulnerability assessment in an organization depends on the deployment methods used and the internal processes in place to act on the reported data.

The value of the VA data is limited unless organizations have a structured approach to deployment and a process to act on the data.

Network VA scanning products are maturing, but require trained staff and frequent use is needed to be effective. Many vendors provide active network scanning, but there are significant differences in delivery, scope, and vendor size.

♦ Delivery: software, appliance, and/or managed service

♦ Scope: network, systems, web server, and/or database

♦ Vendor size: from start ups to tier one companies

Case Study: Vulnerability Assessment


Proper planning, careful deployment, and a detailed follow-through on your action plan are critical to a successful VA tech deployment.

5.1 Proper Planning Planning starts with knowing what is out there for vulnerability assessment.

Know the Approaches, Focus Areas, Reporting and Tools

There are three basic approaches to VA: active network scanning; passive observation of network traffic; and persistent agents on the end node. You should deploy at least two of these three methods for depth and breadth of coverage.

1. Active network scanning, also referred to as network VA will remotely scan (active, turned on) devices over the network without requiring agents.

2. Passive observation of network traffic does not actively scan endpoints, but it captures traffic between endpoints to determine their state based on traffic patterns. (IDS, IPS)

3. Persistent agents reside on the endpoints. They collect information in real time and can determine aspects of the endpoint that are not possible from remote scanning such as applications or services installed but not running.

Different Vulnerability Assessment (VA) focus areas:

♦ System or OS vulnerability assessment

♦ Web app security scanning

♦ Database vulnerability assessment

♦ Wireless vulnerability assessment

♦ VoIP vulnerability assessment

♦ Source code vulnerability scanning

Classification of assets is a crucial element of the risk assessment used to prioritize remediation activities. Classify assets based on the applications they support, the data that is stored, and their role in delivering crucial business services.

Where to focus VA efforts is a business risk issue: In our initial deployment we did not split it up, but we can assign asset groups in the process.



5.1.1 ABOUT US

Our primary job as administrators is to know our assets – is our work correct? Am I in compliance with policies and system integrity? VA helps me know.

We wanted a targeted tool – assessed by an outsider based on SANS known vulnerabilities – not based on our own internal thinking.

The vulnerability seeking worm attacks of 2001, 2003, and 2004 drove us to deploy VA products and services as part of the enterprise security infrastructure.

We have a patch policy and IT governance, but we had no way to declare what to do because we had no way to know what was going on. Now we can check for compliance.

We have delegated remediation. VA is used to self evaluate and remediate; central IT gives departments parameters to work within. Remember, we are a decentralized organization, for better or for worse. This information, however, rolls up to central IT.

Our VA product was selected in 2006 and a contract was awarded to Qualys. We started a project in 2007 to implement the Qualys solution for vulnerability management.

We need to scan during business hours when devices are turned on; scan 15000 nodes in an 8 hour window. This is a basic VA requirement for us.

Our network security strategy: ISO17799 audit, network vulnerability assessment, then penetration testing.

5.1.2 A LITTLE MORE ABOUT US…

What we experienced from 2001 to 2006

The Melissa virus attack told us that we needed a tier 1 virus vendor that automatically updates. We spent seven days getting things under control when this virus came out.

Things improved a bit by the time the ‘I Love You virus’ hit. It took us four days to get on top of this one. Our response time improved. But we knew we needed to be a lot better than that.

We started scanning our environment for certain vulnerabilities with free tools. We wrote custom tools to scan for major viruses: I love you, Slammer, Nachi. But these proved to be limited in scope.

At the same time we centralized our anti virus and mail routing to help mitigate these problems. The end points are still a question though. For example,



someone might bring a home laptop to work and plug it onto the network thereby introducing problems.

2003-2004: we used the MS vulnerability assessment tools and looked at tools from anti-virus vendors. We needed a solution to keep pace. Writing custom code to scan was no longer effective enough for us. Adhoc security was not secure enough. We needed a cohesive, collective approach. We did put in place a patch management policy, but it had no teeth and we had no way to enforce it.

2005 – 2006 we did an assessment of our needs, wrote an RFP, and selected a vendor for a vulnerability assessment capability.

5.1.3 STARTING TO SOLVE THE PROBLEM

1. Know what you have: what systems are out there.

2. Know what you have: what vulnerability assessment products do you have?

3. Do you want to just evaluate for vulnerabilities? Or do you want a product to both evaluate and mitigate vulnerabilities?

4. If you find a vulnerability, do you let the system handle it or do you manually remediate?

5. Know what your policies are before you start scanning – what do you have and what will this do? Refresh your policies as needed.

5.1.4 PLANNING CONTINUES WITH KNOWING WHAT YOU NEED

“Do I patch and scan?

or do I scan and patch?”

“How do I use the investment I already have?”

Does scanning drive my security business or… do I assess the work I’ve done?

5.1.5 REQUIREMENTS OF A SOLUTION

1. A single device or element may be vulnerable to anywhere from 10 to hundreds of unique vulnerabilities and exposures; however, generally only a small set of remediation activities can be taken to eliminate the root cause of a vulnerability.

2. We wanted a network-based approach that could accurately discover and evaluate vulnerabilities on managed and unmanaged systems.



3. If using a point and shoot solution, then look for products that support enterprise level requirements such as centralized reporting and admin, role-based access control, integration with workflow, and tiered deployments.

The VA choice: Do we choose a strong assessment tool? Or do we choose a tool that does both an assessment and remediation?

We went with a higher scanning capability from a vendor that specializes in vulnerability assessment. Because…

1. We are a decentralized organization – departments have their own tools and needs

2. There are overlapping products in this space – WSUS, SMS

5.1.6 THE NETWORK VULNERABILITY ASSESSMENT (NVA) VENDOR LANDSCAPE:

♦ 2 large vendors (IBM/ISS and McAfee) sell VA technology and integrate it with related security products

♦ One vendor (Qualys) is focused on delivering VA as a service.

♦ 7 smaller point solution vendors provide a mix of software, appliance, and / or service based offerings.

The vendor situation introduces a viability risk for the smaller vendors in the market.

A bit more about the vendor that met our requirements

1. We are using Qualys. It delivers VA scanning and on-site appliances as a service. Qualys scanners can also be deployed as managed appliances.

2. The company receives high grades from customers on its level of support and responsiveness.

3. Qualys also offers a central database that can be deployed at the customer site.

4. Optimal use for Qualys: if you need VA but do not want to invest in the internal resources need to support a product deployment.

5. Qualys can find rogue nodes in the network as opposed to agent products. This is one reason why we chose it. However, this works as long as the devices are turned on when the scanning is happening.

6. And Qualys provided free, onsite training. A real plus for us.



5.2 Careful Deployment Qualys will find rogue devices, but they could still connect to you without being found by Qualys.

Current functional challenges in deployment: volume and language of data output, contextual knowledge of the element being assessed, disparate assessment technologies, and a breadth and depth of coverage.

VA tools can generate an overwhelming amount of data. It can be a challenge to get quality report output.

Consider separation of duties: admins don’t audit themselves.

Remediation activities to eliminate root cause of a vulnerability or exposure:

1. Modify user access or rights

2. Patch or upgrade the vulnerable code

3. Uninstall vulnerable application

4. Reconfigure system settings (disable services, close ports, disallow protocols, etc.)

5. Modify the network connectivity or network access to the vulnerable device

6. Strengthen password and authentication

Use differing scan profiles that imitate light to heavy attack patterns

5.2.1 DEPLOYMENT CONSIDERATION AND IMPLEMENTATION TIPS

Best practices for deploying NVA tools:

1. Vendor communication

2. Initial testing in a pre-production environment scanner placement

3. Prepare operations team

4. Perform initial scans

5. Tune scanners

We’ve scanned a majority of our devices so far. It takes 8 seconds per device.

Installation and impact could be major.



Questions to ask during testing:

1. Does the NVA product cause any application instability during a scan of an end device?

2. Does it cause any services or devices to fail?

3. Does it consume too much network bandwidth?

4. Does it create excessive log entries?

Here’s what we found on initial scans:

1. Unapplied patches

2. Unnecessary services running – FTP, Telnet, Web services

3. Poor security – no password, easy password, service and admin passwords simple

4. Misconfigurations

When initially implementing VA and security configuration mgmt baselines, you’ll discover that the majority of your systems contain many vulnerabilities and security configuration errors. There is typically more mitigation work to do than the resources available to accomplish it.

We apply vendor patches monthly, so we plan to scan monthly. However, this could change.

Once 90% of endpoints are taken care of, then we need to keep up with the new additions and our risk level should drop.

5.2.2 OUR DEPLOYMENT

“What am I scanning?”

“Why am I scanning?”

“How am I scanning?”

“What do I need to be concerned with?”

Ten scanners strategically placed on the county Networks:

Based on pathways, bandwidth for scan traffic, load, hops, time it takes to scan, and where our firewall segments are.

Scanners collect data. Scanners send data to Vendor’s secure database.



County department administrators connect to our Qualys account with secure browser session.

Our Qualys account

♦ We create user accounts

♦ We setup IP scanning ranges and segments

♦ We assign those “asset groups” to a user account

♦ Users (Administrators) can then create “Scan Jobs”

• Scan jobs are flexible

• Scan for All vulnerabilities on certain machines

• Scan for certain vulnerabilities on All machines

• Scan for all vulnerabilities on All machines

• Scan once, or repeat on a fixed schedule,

• Etc, Etc, Etc.

Reports are generated from the scans

What we do with the report information (remediation) will be a policy consideration and be based on our Administration Model.

Our project will create policies that will be presented to our IT governing bodies for review, input, and adoption.

The Challenge: what do you have and where is it?

5.2.3 DEPLOYMENT: SPECIFIC CONSIDERATIONS

To perform the scan, packets are sent to the device on the ports targeted in order to elicit a response.

The level of Brute force password checking is selectable, but it does try to authenticate on all but the “none” setting

♦ Event logs will show attempt

♦ If max incorrect passwords option is set at the box, lockouts could occur. This must be taken into consideration. What’s the impact?

♦ Unit manager (Department Administrator) that created the scan job is the one the sets the level



Scan objective is to discover network condition of the box; the data within the box is unimportant. Files are not checked or read.

What if things go wrong?

♦ Individual administrators must be onboard and ready.

♦ No way of telling what will break: old devices / vulnerable devices

• Only experience is our guide

• No impact so far

♦ Scans can be created to minimize potential risk, but at some point, the scan must happen to identify any impact.

♦ White listing is possible, but vulnerabilities need to be corrected

Timeline

♦ Strategy is to 1st determine impact from scanning

♦ 2nd will be to scan for vulnerabilities

♦ 3rd remediation

♦ Usage policy and ongoing support

Impact:

Start small and controlled, move up to larger and more complex scans

Testing parameters: Light TCP, SANS20, minimal brute force, not authenticated, exceptions allowed.

1. Pilot departments – desktop and servers

2. Server Farms (complex and sensitive) = 500

3. All of County - For discovery, timing, and overall impact

Then we will do it all again, getting deeper, and producing more actionable results.

County’s total IP space is carved up into segments that are assigned to the various County departments.

Top level Application Managers entered these blocks into our account.

Top level Application Managers create user accounts



The collection of IP Blocks for each department are assigned to a Business Unit. The Business Units is assigned a Unit Manager (above)

A County IT Department Administrator (who is now a Unit manager in charge of a collection of IP ranges) can create scan jobs, run reports, etc.

These scans can only be run against their own assets (not cross department)

Unit Manager is the person who will act upon the results of the scan

We found a need for a County wide scan or scan process to ensure the safety of the entire Network. These will be driven by policies (as yet to be created)

♦ Zero Day vulnerability

♦ Critical vulnerability scanning (compliance scan)

♦ Audit scan: did departments scan and patch according to policy

This group will also monitor the overall heath of the system

Scan Description

Build the Scan Job (Unit Mangers)

Choose the asset groups (IP ranges) to be scanned

Choose the profile to use;

TCP Ports to scan

Preset lists or custom selection

Light = Common TCP ports (quicker)

Standard = 1800 ports

Full = All TCP ports (takes longer)

UDP ports to scan

Choose Vulnerability Pattern

RV10

SANS20

FULL (All 5000+)

Customer

Single Vulnerability of your choosing

Set “Password Brute Force” level

None

Minimal (empty passwords)

Limited

Standard



Exhaustive

Use Authentication?

If yes, then provide credentials to the platform.

Scan process will log in and check for many more vulnerability signatures.

Advanced Options

List of TCP ports to exclude from any list selection

Set performance levels for scanning

Etc., Etc.

We scan only a sample of our VOIP phones. We have a management system to upgrade and patch software for the phones.

5.3 Detailed Follow Through Detailed follow through requires policy, procedures, and action.

Your organization must implement a process to prioritize the mitigation of vulnerabilities discovered through VA and prioritize your response to security events.

Prioritization is based on risk to the business.

There are four variables to evaluate when prioritizing remediation and mitigation activities are:

1. The nature of the vulnerability (exploit potential and level of system access if exploited)

2. The external threat environment (published exploit code, published exploit automation, mass or targeted attack activity)

3. The ability to shield the vulnerable asset from exploit

4. Business use of the application or data and value of the business data. Assign a threat / risk level to data.

Our mitigation plan: a security committee will determine severity level of vulnerabilities and the time frame for patching or removing the port/segment from the network.



6. UNIVERSAL LESSONS LEARNED Lessons learned that increase the value of vulnerability assessment.

6.1 Vulnerability Assessment Activities Lessons Learned: These activities increase the value of VA activities

1. Use multiple forms of scanning

2. Tune the deployment

3. Scan frequently and targeting new vulnerabilities

4. (VM life cycle) Classify assets, Prioritize remediation and mitigation activities, implement a repeatable, consistent, accountable workflow

5. Use VA data to support compliance needs

6. Know when to outsource some or all of these functions

7. ACT upon the data to achieve maximum value from network VAs

6.1.1 MULTIPLE FORMS OF SCANNING

1. Active network scanning: eEye, ISS, McAfee, nCircle Qualys, Rapid 7, and Tenable and open source utility Nessus

2. Passive observation of network traffic: (scope of scanning is limited and less accurate than other methods) Sourcefire and Tenable. Net behavior tools also provide passive traffic capture and some basic vulnerability and policy compliance – Arbor, Mazu, Lancope, Q1

3. Persistent agents that reside on the endpoints run with privileged access and collect detailed and accurate information: Altiris, Bigfix, Citadel, Configuresoft, Ecora, LanDesk, NetIQ and Symantec. They generally provide an assessment against a narrower set of vulnerabilities compared with network oriented VA scanners. And… the unmanaged elements on a network that do not have an agent pose the highest risk to an organization.

Recommend: active network scanning should be utilized as the primary assessment method and credentialed access should be used to improve accuracy and depth.

Universal Lessons Learned


6.1.2 DEPLOY CORRECTLY

Deploy correctly or these tools can drown your security staff in false alarms and irrelevant information or unnecessarily burden network resources.

One of the most important factors in deploying VA tools is their placement in the network. Network VA tools are constrained by time and space. You need to determine how large an IP block a single scan engine must assess and in what time frame that assessment must be completed.

Assessment accuracy can be affected by the configuration of the scanner as well. The tool can offer VA data from simple identification of open ports and the base OS to deeper level scanning that will assess aspects of the endpoint to obtain info on all ports, protocols, services, application versions, OS, vulnerabilities and exposures.

Initial scans should be limited in their scope; however, once a level of comfort is reached, it is advised to allow for a deeper level of scanning.

6.1.3 SCAN FREQUENTLY AND TARGET NEW VULNERABILITIES

Networks are dynamic. It’s in a constant flux. Vulnerability data can become stale quickly. It is important to frequently assess the environment using the most up to date vulnerability info. When new vulnerable conditions are made public the question is, “are we vulnerable to this new condition, and if so, what steps should we take to mitigate the risk.”

Run VA scan on a frequent basis (weekly) and augment with targeted scans.

6.1.4 VULNERABILITY MANAGEMENT LIFECYCLE FUNCTIONS

VA assessment efforts are effective in improving security only when the assessment is oriented to business risk and when the assessment is used to drive remediation activities. For VA data to be useful in driving remediation activities, reports must be organized by prioritized remediation activity (install this patch, close this port, change this setting). Unfortunately few VA tools provide this reporting orientation. Most organizations should expect to do manual post-processing of VA output if they intend to use it to drive remediation activities.

6.1.5 SUPPORT COMPLIANCE NEEDS

Report against security policies and controls.

NAC tools protect the network from infection by quarantining infected or noncompliant hosts. The majority of NACs require agents. However there is still a percentage of machines whose state can be determined only through a



remote network scan. You should include network-base VA scanning tools to determine the state of unmanaged nodes as part of the overall NAC infrastructure.

6.1.6 DON’T USE VULNERABILITY ASSESSMENT AS A MANAGEMENT TOOL

Don’t use VA as a management tool – use the management tools themselves: SMS, Altiris, Symantic, MOM

6.2 Ten Recommendations to Limit the Probability of a Successful Attack

1. Perform Vulnerability Assessments frequently and act on the data

2. Include NVA scanning tools as part of the overall network access control (NAC) audit process – NAC is the best way to deal with unmanaged nodes accessing the network, to limit the spread of infection from one of the devices, and to limit network disruption.

3. Use a third party provider to perform a thorough penetration test annually – pen test is different than VA. VA attempts to ID the state of the endpoints against a database of known vulnerabilities. Pen test attempts to ID all potential exposures throughout the IT environment or system

4. Use external threat intelligence and update defenses as soon as exploit code is identified.

5. Expand VM activities to include net devices, enterprise apps, database apps, as well as internally developed web apps. – most organizations focus VM activities on Microsoft desktops. Unfortunately, there has been a dramatic increase in critical vulnerabilities against networking devices (Cisco), database apps (Oracle), and Web Apps.

6. Expand the VM process to shield the environment in the face of critical threats as opposed to rapid patching

7. Always update security defenses with the latest signatures, updates and configuration changes – security technologies should be updated with the latest signature and configuration files weekly or bimonthly.

8. Continually review and update the desired state of devices on the network based on the internal security posture and policy.

9. Integrate identity and access management to identify any patterns of activity that indicate suspicious behavior that can lead to an incident.

10.Use network behavior analysis (NBA) tools to detect suspicious behavior on the network that may indicate an impending attack.



6.3 Considerations As We Move Further Along With Implementation

What’s ahead for us as we continue with implementation?

We will reassess in a few years if we need the same tool or not. Now we are working to identify what we have (endpoints), who owns them, and where they are. We really don’t know the overall risk level yet.

With proper management and with proper policy with teeth, we can get risk to flatten out with smoother ups and downs.

After first scans:

• address managing the white list,

• department administration (they scan and maintain their own devices),

• county global compliance scans,

• Policies to be developed:

• Usage

• Escalation

• Audit

• Remediation expectations

• Roles and responsibilities

And then:

• IT resource classification

• Creating resource groups based on business use

• A native capability to classify assets discovered by the security software

• Workflow and incident management

Organizational structure and separation of duties dictate that security teams responsible for auditing the environment are not responsible for resolving the vulnerable or noncompliant conditions.



6.4 Costs I am sure you are wondering “What’s covered and what does it cost?”

Costs depend on the type of solution acquired.

Buying a vendor service

• Cost is by subscription and probably will be based on an annual node count.

• Licenses may be needed for each node or by server.

• Scanning 15,000 nodes could cost on the order of $150,000 a year.

• Be prepared to answer the question, “how long do I pay for this?”

Bring the vulnerability assessment completely in house

• There is a large one time charge up front. The up front cost could be on the order of $500,000.

• There is a need for ongoing staffing to support and manage the system.

6.5 For Us, It is Time for Action! Start with a network vulnerability assessment capability.

The first step in our vulnerability management process is a network vulnerability assessment capability.

Your questions should be well on their way to being answered now.

1. How do we get management to understand the importance of fixing vulnerabilities as soon as possible?



4. Do I want vulnerability testing, penetration testing, a security plan, etc?

5. What should our department’s technical staff be doing to test for vulnerabilities?

6. What exactly is vulnerability testing?



7. How do I write the functional requirements so the vendor will know what I want done?

8. What products are out there for vulnerability testing? How much do they cost? How much work is involved in setting them up and monitoring them? How many staff does it take to monitor it?


6.6 The Next Strategic Issues for Us… We will continue to improve the existing process, but we are already looking at the next serious issues.

1. Network Access Control – the worm attacks of recent years created a demand for a NAC process that would prevent corrupted and dangerous systems from gaining network access. NAC efforts improve defenses against corrupt and dangerous systems by interrogating a node as soon as it is plugged in.

2. Security Metrics – there is a growing need to be able to quantify the effectiveness of our security program. We need a way to measure our overall security posture, as well as the effectiveness of our security program. We are considering two ideas to roll up a measure of our information security confidence level and our information security risk posture.



6.7 Timeline, Prerequisites and Co-Requisites of Security from the Business Side

Recapping our security program approach

1. Countywide IT Governance - 2000

2. Countywide IT Strategic Plan - 2003

3. Assessment needs & requirements – 2005

4. Security Program – 2006

5. Procurement – 2006

6. Project planning and deployment - 2007

♦ Strategy is to 1st determine impact from scanning

♦ 2nd will be to scan for vulnerabilities

♦ 3rd remediation

♦ Usage policy and ongoing support

What could you expect if you go and do likewise?

Risk

Time



7. CONCLUSION Will YOU be able to capitalize on this case study? Opportunity is knocking at the door.

7.1 The Crisis Things are reaching the boiling point. The pot is on the stove. The heat is turned up. The central problem has been defined.

Privacy and security breaches threaten our stability. You need to be anchored with a well-grounded security program. The crisis of the situation is front and center.

7.2 Vulnerability Assessment Assess and remediate vulnerabilities before they are exploited.

Is it just something that is nice to do?

Or is it important, but you hope someone else is taking care of it?

Or is it essential to your business growth and stability and you have a stake in it?

Conclusion


…helping you get the most out of what you put into protecting information

Contact information:

Jim Reiner, Information Security Officer

[email protected]

County of Sacramento – www.saccounty.net

916-874-6788

Network Vulnerability Assessment and Mitigation Case Study

2007 County of Sacramento

This publication may be stored or reproduced in any way you find helpful. Just give credit to us.

The authors and the County of Sacramento have made their best effort to produce a high quality, informative and helpful book. But they make no representation or warranties of any kind with regard to the completeness or accuracy of the contents of the book. They accept no liability of any kind for any losses or damages caused or alleged to be caused, directly or indirectly, from

using the information contained in this book.

Screenshots in this book are directly from publicly accessible file archives. They are used as ‘fair use’ under 17 U.S.C. Section 107 for news reportage purposes only, to illustrate various points

that are made in the book. Text and images available over the Internet may be subject to copyright and other intellectual rights owned by third parties. Some are copyright

Gartner Group and used by permission.


Sacramento, California

USA 95814

Conclusion

http://www.saccounty.net/

county of sacramento gtc west 2007 information security

Documents