YOU'VE BEEN

HACKED!

Michigan Summit

You have been hacked now what?

What should you do?Organizations must assume they are compromised, and therefore, invest in detective capabilities that provide continuous monitoring for patterns and behaviors indicative of malicious intent.

Define an incident response procedure that details the roles of appropriate business and IT contacts throughout the organization and other departments needed to respond to security incidents, including human resources, public relations, legal and executive management.

What happened and how we handled them will follow.

What do they have?

Web Applications Breach

In 2013 just after 6 months of being in the role of CIO, we had what appeared to be a potential very large data breach.

I was off work, driving to Detriot, when I received a phone call from one of our department managers. It started off with: I am not sure if we should be concerned but I think we may have a problem. She started telling me about an applicant they interviewed for their data analyst position of their health care program. The applicant showed them some screen shots of their application and database that the person was able to access with what they called simple SQL injection commands.

My first question was what application is this.

Then I asked what information did the person claim or show you:

I now know why they tell you not to talk on your cell phone while driving.

She stated the applicant showed / had PI information, Name, Address, Date of Birth and SS#. The application has over 300,000 records with this type of information.

Secure the systemI hang up the phone, called our Database Admin and Network Admin and told them to take the application off line ASAP like yesterday!

I had the DB Admin start looking into the what happen, how it happened and what he can find out and the same with the Network Admin.

We also had them copy the application and start working on securing it so we can get operations back to normal.

At this point, I still didn’t know what the person really had and we didn’t have expertise in house to truly determine what they had.

Determine what they gotThe next phone calls I made was to the State Department of Technology Management and Budget Security Office and Michigan State Police Computer Crimes unit.

Of course, they wanted to know who the person was. Since we knew that information we provided that information.

Then I find out the person is not a U.S. Citizen and MSP start working with my staff to determine what the person got!

The FBI came in as well since we are now talking about a non U.S. Citizens.

Can it get worse or better?

Get operations back upWe have the contractor that wrote the program for the Health Department without IT’s input re-write and secure the application with the IT department’s expertise.

State Police interview the individual and took their computer equipment.

We got the application secured and had it pen tested and brought the application back on-line.

State police forensic the equipment and found the person only took a couple of screen shots of the database and printed them to bring to the job interview. They ended up not having all the sensitive data.

Did we do it rightThe person did not see anything wrong with what they did and had no ill intent. The person was trying to show what they could fix what was wrong with our system. They exceeded at getting our attention since this did expose a huge concern for us.

We now have a notification process defined: what to do, when to do it and who to notify (most taken out of what we did and formalized it).

The question from some was did we over-react bringing in MSP and the FBI?

I would rather over-react in this type of situation than not. If there was ill intent and they had all the data it would have cost over a million dollars.

What we do now is assume that we are already infected. We changed our "incident response" mindset to a "continuous detection and response" process.

We have invested in tools and processes that can detect malware infections that have evaded traditional blocking and prevention solutions.

We have to implement preventive controls to "harden" endpoints. These are highly effective and should be revisited by looking at new tools that are emerging to simplify this task regularly.

We have to ensure business impact analysis aligns business continuity and IT disaster recovery plans to the value of the business processes being protected. This also helps address IT complexity by supporting IT application and data classification.

Bottom line everyone agreed that it is better to assume the worst and hope for the best.

What we learned and did

Define an incident response procedure that defines the roles of appropriate business and IT contacts throughout the organization and others needed to respond to security incidents, including human resources, public relations, legal, law enforcement and executive management.

Retain either internal or external resources for executing an incident response plan: specifically target resources with digital forensics and malware analysis knowledge.

Security incidents should remain confidential within the incident response process and proper workflows as well as collaboration need to exist between involved parties during execution of the incident response procedure.

The number one item was to secure the data first! Determine what and how the breach occurred then bring the systems back on-line once secured.

Once you know how the breach occurred work on redefining your tools used and change processes and/or procedures to prevent future ones from occurring.

Your success or failure will not depend on if you got breached but how you handled IT

Second Incident

Ransomware or Crypto locker virus

Earlier this year, we got a call on our afterhours number of a potential virus issues at our Health Department around 5:45pm. Of course I find this out as I am heading into a board meeting.

The Network admin talked to the end-user who said she had a couple of files that were encrypted and she had to contact the company to get her file back.

We started our incident response plan at this point.

Crypto Locker virusThe first thing we did was to disable the user’s account and the infected computer so her credential and the laptop was unable to infect any further data.

We then looked at what the user had rights to and checked the files.

This is where we discovered that over 1000 files were affected across several directories on the SAN.

Get things back to normalWe moved into a DR process to bring the business up. We started running full scans on all volumes of the SAN and servers. We investigated the time frame of corruption and started removing the corrupted files.

We took the infected computer to have an analysis on it to find out how this happened. We provided the laptop to MSP for their help.

Once we got business back into operations, we then went into forensics mode to find out the how and why.

From the time we were notified of the potential issues to the time we had operations back to normal was within 1 hour and business was able to continue otherwise.

How did this happenIn this phase, we discovered the person had Dropbox installed on their laptop. They were also using it to access personal email from the web.

Since the laptop was not always on the network, the AV software was outdated. These all lead to the infected file getting on to the laptop and going wild on the network.

We reimaged the laptop, we then started a full force check on all systems to verify that the AV was installed and up-to date.

We implemented some better management protocols and tools to ensure our Antivirus software is and continues to be updated.

We are looking at other tools to put in place to protect the network from the inside so the network damage can’t continue beyond the one infected machine.

While over 1000 files were affected there was minimal impact to the county and it only cost us some staff time no real dollars.

We have now implemented a security awareness monthly training for all users.

Issues was reported at 5:45. By 6:00 PM, the affected user and system was isolated.

By 6:15 the laptop was in IT’s possession and restoration efforts were started.

By 6:45 the files that were corrupted/encrypted were restored to their previous state.

How did we do itWe were prepared and knew what to do.

The IT staff informed the correct staff and the correct time to minimize the affects.

We have DR plans and a DR SAN which allowed for the quick restoration of the files.

The only comments we got from the board and the department were simple.

How can we prevent it in the future?

How were you able to get everything done so quick?

Again it was not an issue of the fact we got hacked it was more focused on what we did and how quickly we did it. Confidence level of IT operations continue to grow as they know we can handle critical situations.

Time is money and having IT services down cost money and frustrations not only for the users but the citizens we serve.