ics cyber security effectiveness measurement · business thinks about cybersecurity, but in its own...

Alexey Lukatsky

Security business development manager

ICS Cyber Security Effectiveness Measurement

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Not Petya / Nyetya Tools

Tactics• Supply chain and victim to victim pivoting

• Rapid Infection Spread

• Destroyed Countless Systems / Networks

Processes• Designed to inflict damage as quickly and

effectively as possible.

• Appears to be Ransomware, but is purely destructive

• Wormable Ransomware

• Designed to Spread Internally Not Externally

• Leveraged Eternal Blue / Eternal Romance and Admin Tools (WMI/PSExec)

• Advanced Actor associated with a Nation State

• Destructive Attack Masquerading as Ransomware

• Most Expensive Incident in History

Description

ICS Kill Chain

1

2

3

4

5

6

7

8

ConfickerAPT1

Иран vs

США

BE3

HAVEX

Stuxnet

Ukraine

2016

WannaCry

Neytya


Why we need to measure our effectiveness?

• Good security not visible

• We want to show that we work well

• Top management often wants to compare itself with others

• We want to see the dynamics


Rare Remote Possible Likely Very likely

Catastrophic 6 7 8 9 10

Significant 5 6 7 8 9

Moderate 4 5 6 7 8

Minor 3 4 5 6 7

Insignificant 2 3 4 5 6

Accept(score = 2,3)

Monitor(score = 4,5)

Manage(score = 6)

Avoid / Resolve(score = 7)

Urgently avoid/ Resolve(score = 8, 9, 10)

“Best practices” for security measurement

• Not specifically, not quantitatively, conditionally…

Impa

ct

Probability

Cybersecurity is state of protection of the interests of enterprise stakeholders in the

information area, determined by the totality of balanced interests of the individual, society,

state, and business

Or process? Not

important!

Efficiency/effectiveness is the quantifiable contribution to the achievement of ultimate goals


What goals can we have?

• Fulfillment of NERC CIP or ISA/IEC 62443 requirements

• Categorization of all CI objects

• Certification of key processes for ISO/IEC 27019

• Reduce the number of ICS cybersecurity incidents to 3 per month

• Implementation of secure remote access to ICS for contractors

• Reduce downtime from ICS cybersecurity incidents to 2 hour on average

• Cost reduction for ICS cybersecurity for 15%


• Operational (наиболее привычные)

• Realtime, day-to-day

• Logs, rules, signatures, etc.

• How effective is your security measures?

• Tactical

• Change control

• Scorecards and audits

• How effective is your security program?

• Strategic

• Corporate risk and business alignment

• How are we secure?

Strategic

Tactical

Operational

Measurements are different


Tactical metrics examples

• Incidents requiring manual cleanups

• Mean-Time-to-Fix

• Also TTR (Time-to-Recovery) or TTC (Time-to-Contain)

• Mean-Time-to-Detect

• Mean-Time-to-Patch

• Involvement of staff in cybersecurity activities

• Mean cost to mitigate vulnerabilities



• % of ICS without known severe vulnerabilities with CVSS >7.0

• % of changes with security review

• % of changes with security exceptions

• ICS cybersecurity budget allocation (% of total, IT, cybersecurity, ICS)

• Compliance rate

• Cost of incidents



• Time between creating and closing a ticket for an incident

• Ratio of open and "closed" incident reports

• Ratio of incidents and tickets

• Number of repeat incidents

• Ratio of communication methods (e-mail / calls / portal)

• Number of false positives (non-existent incidents)


SMART principle for metrics selection

• SMART – Specific, Measurable, Achievable, Relevant, Timely

• As concretely as possible, without double interpretations, for the right target audience

• The result should be measurable, not ephemeral

• Why choose a goal that is unattainable?

• Relevance to goals

• Timeliness and relevance


SMART usage example for ICS Cybersecuirty

Characteristic Example of bad metric Example of good metric

Specific The number of failed login attempts to

the HMI

The number of failed login attempts

to the HMI for one week for one

employee

Measurable Income from the implementation of an

ICS cybersecurity

The employees loyalty level about

ICS Cybersecurity

Achivebale The absence of cyber security

incidents in ICS for the current

quarter

The number of ICS cybersecurity

incidents in the current quarter <5

Relevant The number of opened projects for

ICS cybersecurity

The number of completed on time

projects for ICS cybersecurity

Timely The number of patched ICS nodes

last year

The number of unpatched ICS

nodes current year

How to move from hundreds of operational metrics to one or two

strategic?


From individual metrics to measurement program

• EPRI (Electric Power Research Institute) Research Program

• Creating Security Metrics for the Electric Sector (Parts I, II, III, IV)

• Applicable to a wide range of industrial enterprises outside the electric power industry

3 strategic metrics

10 tactical metrics

45 operational metrics



Strategic Metric Name Tactical Metric Name

Protection Score Network Perimeter Protection Score

Endpoint Protection Score

Physical Access Control Score

Human Security Score

Core Network Vulnerability Control Score

Core Network Access Control Score

Data Protection Score

Security Management Score - Protection

Detection Score Threat Awareness Score

Threat Detection Score

Security Management Score - Detection

Response Score Incident Response Score

Security Management Score - Response



Tactical Metric Name Operational Metric Name

Network Perimeter Protection Score Mean Access Point Protection Score

Mean Wireless Point Protection Score

Mean Internet Traffic Protection Score

Mean Count-M Malicious Email

Mean Count-M Malicious URL

Mean Count-M Network Penetration

Security Management Score - Protection Security Budget Ratio

Security Personnel Ratio

Cybersecurity Risk Tolerance Score



Operational Metric Data input to the Formula

Mean Access Point Protection Score Number of inbound connections per day

Number of dropped inbound connections per day

Number of all alerts per day

Number of security alerts per day

Number of probes per day

Number of confirmed DOS attempts per month

Чnumber of confirmed intrusion attempts per month

Number of confirmed incidents that required human intervention per month


Automation tool: EPRI MetCalc

What does the business think of all these metrics?

Business thinks about cybersecurity, but in its own way

Reservoir

Pump

Water intake

Water treatment

plantsUnderground tank

Pump

Distribution

Cleaning with

reagents, ozone and

coal

Sump

Flats /

Houses

Water

meter

Correct and

uninterrupted bills

Smooth operation

Continuous diagnosis

Telemetry control

Continuous monitoring

FZ-1

52

Order №31 CIP Law

Water supply process


The difference in the perception of top management and cybersecurity / IT / ICS

Cybersecurity / IT / ICS

• Deep dive to details

• Unwillingness to share collected data

• Data for data, not for decisions

• What? Where? When?

Top management

• Bird's-eye view

• Data for decision making

• What will happen? What to do?

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco PublicTime

Productivity

0

20

40

60

80

100

А

В

С

ВТ

Т1 Т2 Т3

D = System failure / disaster

R = The possibility of attenuating or mitigating the effect before or during a

negative event

A = The ability to absorb and degradeВ = Lower limit; threshold value

ВТ = Lower limit duration

С = Ability to return to baseline

D → R

How does a business see security incidents?

Reduce А?

Reduce Вт?

Reduce С?

Reduce Т1, Т2 and Т3?

Let's try to reformulate our goals

Profit increase

Geo expansion

Sales increase

Production optimization

Reduction in logistics costs

Loss reduction

X hours of downtime due to ransomware

Y hours of process downtime due to DoS/DDoS-attack

Z hours of employee downtime due to spam

N rubles fine from supervisory authorities

Business

Cybersecurity


From the “for myself” measurement to the measurement for business

75%

55%

Q2

Q1

The number if incidents

by sources

The number of ICS incidentsDowntime

Incidents dynamics

Contracts loss

$35M127


Cybersecurity incidents loss types

Productivity•Downtime

•Deterioration of the psychological climate

Response

•Incident forensics

•PR-activity

•Support Service

Replacement•Equipment replacement

•Re-entry of information

Fines•Legal costs, pre-settlement

•Suspension of deals

Competitors•Know-how, commercial secrets

•Customer churn, overtaking by competitors

Reputation•Goodwill

•Decrease in capitalization, stock price

Other•Rate downgrade

•Decrease in profitability


Impact categories Insignificant Minor Moderate Significant Catastrophic

Finance impact of more

than $Y

$1М $5М $10М $50М $100М

Let's be more specific and measure the money

• The cost of direct losses from disruption of business operations

• Business Transaction Recovery Cost

• Decrease in stock prices (dumb indicator, but sometimes also measurable)

• Fines

• Lost profit (if you can count it)

• Decrease in customer loyalty

• Replacing equipment or re-entering information

• Interaction with affected customers, etc.


Questions for defining strategic business metrics of cybersecurity

• What will stop or slow down operations in your organization?

• What will lead to a decrease in profits / revenue / margin / market share of your company?

• What will lead to a decrease in the quality of the product / service?

• What will lead to a negative impact on the goal of the company / business unit / business project / executive sponsor?



Outage of more than X

customers

10 customers 100 customers 500 customers 1000 customers 5000 customers

Business operations

disruption of >= Z min /

hours / days

1 hour 4 hours 8 hours 2 days 5 days

Serious injury to >= A

people

0 people 0 people 1 person 10 people 50 people

Breach of data for >= B

customers

100 customers 1000 customers 5000 customers 10000 customers 100000 customers

Loss of >= C customers 5 customers 10 customers 25 customers 50 customers 100 customers

Loss of market share for

D%

0% 0% 1% 3% 7%

Productivity loss for E% 0% 1% 3% 5% 10%

If you can’t count in money?


The duration of an cybersecurity incident in terms of cybersecurity and business

The influence level and price components of an incident changes over time

This illustration can be used to estimate recovery time after an attack

RPO – Recovery Point Objectives, RTO – Recovery Time Objectives, MAD – Maximum Allowable Downtime



Reduction of power

generation by F megawatts

Power reduction is

acceptable

Power reduction is

acceptable

100 MW 1000 MW 10000 MW


Publications in mass media Absent In local consumer print

media

On local TV or in local

industry publications

On national TV or in

national consumer print

media

Highlighted broadcasts or

reporting on national TV or

in national industry print

media

Industry specific metrics

How to measure cybersecurity for a business, but not with money?

Can compare yourself with competitors?

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

План & бюджет

Организация

Защитные меры

Архитектура

Процессы и операции

Осведомленность

Реагирование

Управление уязвимостями

Оценка рисков

Корпоративное управление

В среднем по отрасли

У нас

Tricks: instead of comparing with competitors (if there is no data),

you can compare yourself in different states (there was - now - in a

year - ideal)


5 important metrics

• % of cybwersecurity activities unlinked to business goals

• Number of projects / activities linked to business goals

• % of projects / assets / services that are important for business that do not meet cybersecurity requirements

• For example, uncontrolled remote access by contractors

• % of projects / assets / services that are important for business and whose security measures are inadequate or ineffective

• Or for whom during the incident the response plan did not work

• The likelihood of providing services during an cybersecurity incident

You can still play with the risks ...


Common errors in effectiveness measuring

• Choosing hundreds of metrics instead of focusing on strategic

• Measuring what is easier to measure instead of focusing on measurement goals

• Lack of business focus

• Focus on operational result-oriented metrics instead of evaluating process performance

• Lack of context

• Cybersecurity price reduction with incidents growth

Key Success Factors

• You must understand what you are doing in the field of information security

• You must understand your business

• You must understand your target audience

• You must be able to combine these three elements together

• You need to know where the data is

• You must be able to code/program

A Ne

w L

ook

at C

yber

secu

rity

M

easu

rem

ent

Thank you!

[email protected]

ics cyber security effectiveness measurement · business thinks about cybersecurity, but in its own...

Documents