Incident Handling in Academia

What to do when you have been hacked!

The Presenters

Scott Fendley– BS Comp Science – U of AR 1999– MS Comp Science – U of AR 2004– Security Analyst, Dept of Computing Services– Volunteer Incident Handler, SANS Institute

David Merrifield– Associate Director of Computing Services

Session Description

Explores how to handle the attacks on your Internet infrastructure.

Discusses a time-tested 6 step procedure for Incident Handling.

Touches on the legal issues relevant to all Academic Institutions (K12 or Higher Ed)

Dealing with Law Enforcement and handling Evidence

Employee Monitoring vs Student Monitoring

Disclaimers, Disclaimers, Disclaimers

I am not a lawyer. Consult your nearest legal counsel if you choose to handle incidents on your campus or have questions.

The majority of this information is the basis of my procedures at the University of Arkansas, but your mileage may vary.

Foundation of Incident Handling

An Action Plan for dealing with intrusions, cyber-theft, denial of service and other security-related events

Events can be of a electronic nature or of a physical nature.

Definitions Incident – an adverse event in an information

system, and/or network, or the threat of the occurrence of such event.– Ex: unauthorized use of another user’s account– Execution of malicious code– Unauthorized use of system privileges

Event – Any observable occurrence in a system and or/network.– Ex: Packet Traces– System Boot Sequences– Anything that you can record in your IH notebook

Incident Handling Metaphor

Incident Handling is like First Aid. The Handler is under pressure and mistakes

can be costly Practice is a key. Skills degrade without

use. Use pre-designed forms and procedures,

and call on others for help.

Emergency Action Plan

Remain Calm. Communicate with your management, and

coordinate with your co-workers to keep things focused.

Use formalized language.– EX: Whiskey Five Yankee Mic, We have a

bogey on your nine.– Explicit meaning, no room for interpretation is

less likely to cause mistakes.

Emergency Action Plan

REMAIN CALM (still!) Do not hurry. Mistakes can be costly.

Notes, logs and other evidence are crucial– If the perpetrator is ever found and arraigned, how can

you testify if your notes are not organized and detailed?

Failure to take notes is the most common mistake. Consult your legal counsel for how long you

should keep your logs. Quality not Quantity

Emergency Action Plan

Take good notes.– Remember what your English teacher taught you.

– The 4 W’s• Who?

• What?

• When?

• Where?

– Extra Credit for the 5th W and the H• Why?

• How?

Emergency Action Plan (1)

Notify your manager of your progress Do you have easy access to your School’s phone

directory? Pager numbers? Home numbers? If you are over your head, do not hesitate to ask

for help– FBI Field Office

– handlers@incidents.org

– Local Law Enforcement

– Trained Computer Forensic Investigators

Emergency Action Plan (2)

Enforce a “need to know” policy. Do not tip your hand to potential insider

threats. Use out of band communications. (Don’t

email people about IH discussions.)– Telephones– Faxes– Personal Visits

PGP Keys

Emergency Action Plan (3)

Contain the problem. (stop the bleeding)– Pull the network plug?– Pull the power plug?– Forensic Evidence Quandary.

Containment Micro Example

Call the user and say “Take your hands off the keyboard and move away from the computer.”

Stand up go to the back of the computer and unplug the network (and/or modem).

Don’t touch anything, we’ll be right there. Fax instructions/forms for them to fill out.

Emergency Action Plan (4)

Make a backup of the affected system(s) as soon as is practical. Use new, unused media.

Make a binary, or bit-by-bit backup. Failure to make a backup is the second most

common error. Chain of custody of the evidence.

Emergency Action Plan (5)

Get rid of the problem. Identify what went wrong if you can. Take steps to correct the deficiencies that allowed the problem to occur.

Nuke the computer or just scrub it? Get back in business using clean backups

and monitor the system to make sure it can resume functioning.

Emergency Action Plan (6)

Learn from this experience. Share your experience with others.

– Sys-admin List for K12

– Arktech List for Universities and Colleges

– Another useful list is Unisog@sans.org for all Educational entities.

Review the incident from start to completion. Identify areas of improvement Engineers versus Mathematicians

Seven Deadly Sins of IH

Failure to report or ask for help Incomplete/non-existent notes (Accidental) Mishandling/destroying

evidence Failure to create working backups. Failure to contain or eradicate Failure to prevent re-infection Failure to apply lessons learned

Emergency Action Plan Summary Remain calm, don’t hurry. Notify your oranizations’s management, apply

need to know, use out of band communications. Take good notes (even if you aren’t/can’t

prosecute). Contain the problem Back up the system(s), collect evidence Eradicate the problem and get back to business Lessons Learned

Six Steps of Incident Handling

Preparation Identification Containment Eradication Recovery Lessons Learned

Preparation

Update your organization’s disaster recovery plan to include Incident Handling

Establish visibility and a compensation plan for the team. (Slush fund for food and caffeine for long weekends or evenings of mitigating an emergency.)

Checklists! Emergency Communications Plan

Preparation Key Points

Password Access Conduct training for incident handlers

(War Games) Establish guidelines for inter-departmental

cooperation. Build relationships with techies and sys admins Develop interfaces with law enforcement agencies

in your area.

Preparation - Jump Bag Small tape recorder

– Blank Tapes

Binary Backup Utils– Safe Back

– Ghost

– Encase

Forensic Software– TCT

– Autopsy

– Encase

Small Hub and cables

Laptop (extra batteries) CD’s with clean binaries

– Sysinternals

– Foundstone

– Windows Resource Kit

Call List, Phone book Cell Phone (batteries) Fresh Blank Media

(CD-Rs Floppys, Zip, etc)

Preparation in a nutshell

Policy People Data Software/Hardware Communications Supplies

Transportation Space Power and

environmental controls

Documentation

Identification

Fire Alarm Analogy– Who can pull a fire alarm?– Who authorizes re-entry?

Maintain situation awareness Provide current “intelligence” Correlate information (mailing lists are

great sources for newest worms/viruses or attacks)

Signs of an incident Intrusion Detection system alarm Suspicious entries in system or networking

accounting Discrepancies in logs (Un) successful logon attempts Unexplained, new user accounts Unexplained processes or services running Notification via abuse@ address or phone call Poor system performance Unusual time of usage.

Identification

Initial Assessment “Efficient handling of errors is part of the process” Be careful to maintain a provable chain of

custody. Use the tape record if at all possible to keep notes

for you on what commands you run and actions you do.

Make law enforcement sign for any evidence you hand off to them. Assign a value to it.

Containment

This is where we cross the threshold in which we begin to actively modify the system.

Keep the system pristine Pull the system off the network (or perhaps

the subnet off the network). Load your binaries, set the path Backup the system

Containment

Safely store any backup disks/tapes so that they will not be lost and/or stolen. Multiple copies are best with volatile media types.

Keep a low profile. Analyze a copy of the backup Report to management on progress Are you sure you backed up the media in

question?

Containment

Acquire logs and other sources of information.

Firewalls, IDS Logs Logs from other systems nearby

Containment

Consult with system owners (departmental technical staff)

Change passwords Determine possible other systems that have

potentially had passwords breached. Packet sniffers are easy to install.

Eradication

Is your schools policy to nuke the computer and reinstall with a secured OS, or just clean and secure?

Improve your defenses Perform vulnerability analysis and system

audits. Locate the most clean backup and carefully

install it.

Recovery

Restore from backups if required Be sure you do not restore the malware Secured system? Validate the system and create baselines Test that everything on the system is working as

expected with the owner. Place the final decision on the system owner of

when to restore operations. Monitor the systems

Follow-up / Lessons Learned

Develop a follow-up report– Start as soon as possible– Include any forms you used in identification

step– Details, details, details!

Lessons Learned Meeting Executive Summary Report Recommended Changes to procedures? Additions to jump kit

Legal Issues to Academia

HIPAA– Privacy Rule (2002)– Security Rule (2005)

FERPA (Buckley Amendment) DMCA Patriot Act

Monitoring

Monitoring employees Student Privacy Student-employees?

Law Enforcement Contacts

University Police City Police or County Sheriff FBI (Field office in LR) Secret Service Department of Homeland Security Infraguard Arkansas

More Information

http://www.sans.org/ http://www.securityfocus.com/ http://www.foundstone.com/ http://www.sysinternals.com/ http://www.incidents.org/ http://ists.dartmouth.edu/

Questions?

Contact me at scottf@uark.edu or call me at 479-575-2022.

Also, talk to those in the state and across the nation for specific questions.– Sys_admin@lists.state.ar.us– Unisog@sans.org