www.egi.eu egi-engage recent experiences in operational security: incident prevention and incident...

www.egi.eu

EGI-Engage

www.egi.eu

Recent Experiences in Operational Security:

Incident prevention and incident handling in the EGI and WLCG infrastructure

Dr Linda Cornwall, STFC.

HEPiX Spring 2015

24th March 2015 Linda Cornwall, STFC 1

www.egi.eu

WLCG and EGI

• The (Worldwide) LHC Computing Grid and The European EGI Infrastructure share a lot of the same resources

• Also share Security teams and activities

24th March 2015Linda Cornwall, STFC.

HEPiX Spring conference, Oxford2

www.egi.eu

Contents

• Incident Prevention • Policy definition• Vulnerability handling• Security monitoring

• Incident handling and incidents from the last year

• Evolving the work



www.egi.eu

Security Incident Prevention

• Far more work goes into preventing incidents than handling them• Security Policy definition• Software Security, especially Software

Vulnerability handling• Security monitoring - monitoring for known

vulnerabilities and insecure configuration



www.egi.eu

Security Policy Definition

• Security Policy definition is carried out by the EGI Security Policy Group (SPG)• Defines the behaviour expected from NGIs, Sites,

Users and other participants to maintain a beneficial and effective working environment

• Output is various policy documents• Parties read and sign, so that they know and

understand what they should and should not do• List of policy docs at:

https://wiki.egi.eu/wiki/SPG:Documents



www.egi.eu

Minimizing vulnerabilities in the infrastructure

• Handling vulnerabilities found/reported• Main activity of the EGI Software Vulnerability

Group

• Assessing software for vulnerabilities• Formally and informally

• Preventing new vulnerabilities being introduced • Developer education, awareness

• Considering new software to be used in the infrastructure



www.egi.eu

Software Vulnerability Handling

• Approved procedure (Under Revision)https://documents.egi.eu/public/ShowDocument?docid=717

• Anyone may report an issue• By e-mail to [email protected]• This may be because they have found it in software• Or it may be that it has been announced

• If it has not been announced, SVG contacts the software provider and the software provider investigates (with SVG member, reporter as appropriate)



www.egi.eu

Relevance and Risk

• The relevance to EGI is considered, and what affect it could have

• Then it is risk assessed, and put in 1 of 4 categories• ‘Critical’, ‘High’, ‘Moderate’ or ‘Low’

• If it has not been fixed, target date set• ‘Critical’ 3 days, ‘High’ 6 weeks, ‘Moderate’ 4

months, ‘Low’ 1 year



www.egi.eu

Advisory issued

• An advisory is issued when vulnerability is fixed if EGI SVG IS the main handler of vulnerabilities for this software, or software is in EGI UMD regardless of the risk.• E.g. Grid Middleware, tools developed in EGI

and collaborating projects

• If EGI is NOT the main handler, e.g.linux advisory only issued if ‘High’ or ‘Critical’



www.egi.eu

SVG message – if you find a vulnerability

If it is NOT public knowledge •DO NOT

• Discuss on a mailing list – especially one with an open subscription policy or which is archived publically

• Post information on a web page• Publicise in any way without agreement of SVG

•DO report to SVG via

[email protected]



www.egi.eu

High and Critical Vulnerabilities monitored

• Sites are monitored for ‘High’ and ‘Critical vulnerabilities.

• EGI CSIRT chases sites which are exposing ‘Critical’ vulnerabilities

• Sites may get suspended if they expose critical vulnerabilities and don’t respond

• Respond if asked to by IRTF/CSIRT

• For ‘High’ risk, up to the local NGIs.



www.egi.eu

Vulnerabilities reported during last year

• 42 new entries in vulnerability tracker (RT)• 12 concerned Grid Middleware - 2 critical

(1related to heartbleed, 1 related to perfsonar/cacti) 4 high

• 16 Linux – 3 critical (heartbleed, Shellshock, Kernel) 5 high

• 4 Cloud enabling – 3 high• 6 VO software – 3 high • Others include 1 high



www.egi.eu

Changing types of Vulnerabilities

• Until about 1 year ago most vulnerabilities concerned Grid Middleware

• Now more concerning VO specific software• Including Data Protection issues • VOs take it into their head to ‘monitor’ activities

in a way that is traceable back to user

• Cloud specific software• Less knowledge about this



www.egi.eu

Incident Handling

• Approved Incident handling procedurehttps://documents.egi.eu/public/ShowDocument?docid=710

• Incidents are handed by the Incident Response Task Force.

• Fortunately there are not many • Incident prevention is quite successful



www.egi.eu

Incidents during last year (8)

• Primecoin mining (Policy violation)• Open Hostkey leaking private information• User cert mis-use• Fed Cloud incident

• Due to bad endorsed VM• UI compromised (4 user IDs compromized)• Shellshock related compromises to Perfsonar nodes

(multiple sites)• Compromise due to port left open• DDoS to some EGI services



www.egi.eu

Evolving the Security Work

• Evolving the security work is necessary due to e.g.• The EGI federated Cloud

• Changing responsibility model• Changing technology

• Long Tail of Science• Different trust model

Have some H2020 funding for EGI engage to carry out this evolution



www.egi.eu

Policy documents under revision

• Getting rid of ‘Grid’• Policies apply to all technology and services

• Acceptable use policy• External draft – request for feedback and

comments

https://wiki.egi.eu/wiki/SPG:Drafts:Acceptable_Use_Policy_March_2015

• Security Policy for the endorsement and operation of Virtual Machine images• Especially for Fed Cloud experience



www.egi.eu

New policy documents

• Data Protection Policy• Formerly only had “Grid Policy on the handling

of User Level Job accounting data• Finding Data protection policy needed as User

level data is being monitored and exposed inappropriately.

• Long Tail of Science Policy• Related to allowing access other than by large

VOs, IGTF certificates• User sub-proxy.



www.egi.eu

Vulnerability handling evolution

• Now more software is coming into use where SVG members have no knowledge• New members of SVG who know about cloud

software, especially tools written within the community

• ‘Expert’ contact for all software Cloud enabling software deployed in the Fed Cloud

• VO software – assume VO security contact is responsible and know who to contact

• No more than 2 steps to the right person.



www.egi.eu

Software security checking

• For some community cloud enabling software have a detailed ‘Technology provider’ questionnaire

• For other software propose something simpler:-- • License details• How long will it be under security support?• How are security problems reported? • Are security problems announced? • Check compliance with Data Protection policy• Some other simple technical checks – e.g. is user input

is validated, bad constructs – not obviously bad



www.egi.eu

Incident response evolution

• Changing responsibility model in the cloud will mean changes to incident response.

• A lot of work is going on including traceability – See Ian Collier’s talk



www.egi.eu

Questions??



www.egi.eu24th March 2015

Linda Cornwall, STFC. HEPiX Spring conference, Oxford

23

www.egi.eu egi-engage recent experiences in operational security: incident prevention and incident...

Documents

hepix spring conference

software provider

new software

egi svg

security incident prevention

egi security policy

operational security

security teams