presenter: richard dolewski - lisug · primary assets needed to operate information systems:...

© 2016 Denovo. Confidential Trade Secret.

Presenter: Richard Dolewski

Send an Invitation & connect on LinkedIn

Receive Tip & Techniques Monthly

Industry Best Practices to improve your

Business Continuity so you: Don’t Fall with the Fallen!

www.linkedin.com/in/richarddolewski

[email protected]

Richard Dolewski

www.linkedin.com/in/richarddolewski

[email protected]

IBM Power Theater

“Nothing on Earth

can prepare you”

The Need of Disaster Recovery

Definition of a Disaster

A sudden, unplanned event, that causes great damage or loss to an organization

The time factor determines whether an interruption in service is an inconvenience or a

disaster. The time factor varies from organization to organization.

DC Outage Costs

Data center downtime continues to negatively impact enterprise profitability and productivity.

The average cost of an unplanned data center outage is about $7,900 per minute

Will your DR plan be enough to save your sinking ship?

DR Plan?


DR Planning is an Ongoing Process

Business

Requirements Plan

ImplementTest

…to be prepared, you must regularly

go through the cycle


Change Control- Best Practices


Drivers of Change

• Have you kept up to date with your IT integrations?

• DR Planning not implemented into your change control

• You no longer have qualified staff

• Expectations of your DR Plan are unrealistic

• Optimistic - In Reality - Not prepared


The Products of a DR Plan

Who Will execute recovery actions

What Is needed to continue, resume, recover or restore business functions

When Business functions and operations must resume

Where To go to resume corporate, business and operational functions

How Detailed procedures for continuity, resumption, recovery or restoration

CLASSIC: WHO - WHAT - WHERE - WHEN - HOW


Downtime is no vacation

Understanding RPO & RTO

Recovery Objectives

Days MinsHrsWks Secs

Recovery Point

Mins DaysHrsSecs Wks

Recovery Time

Recover data to last known

good point before outage

RECOVERY POINT OBJECTIVE

Length of time it takes to resume

to near normal business operations

RECOVERY TIME OBJECTIVE

Traditional Tape Recovery Solution

RTO Recovery Time Objective48 hours RTO – Systems Availability

RPORecovery Point Objective24 Hours RPO - Tape

Are You Aligned With Your Business?

• Management shocked that the IT folks lost all orders from previous day - 24 hrs. of lost data

• Application access is critical

• 48 hour RTO – who agreed to this?

• Do you have new Senior Management? New Owners ?

What About Your Email?

• Email has become mission critical

• Email represents the way you communicate with customers, partners and employees

• In a disaster situation, this may be a key component of your communications strategy

• Is email listed as a critical application?

“If you build it,

they will come.”

The Importance of Backups

Protect Critical Assets

Primary assets needed to operate information systems:

– Hardware and networks can be replaced

– Facilities can be rebuilt or relocated

– Your Data is Priceless!

– YES People are assets too!


Tape Backups - Pros

• Easy to Run

• Easy to Manage – Tape Management Products

• Easy to Automate – Automation Solutions

• High Speed – Reduce Backup Windows

• Capacity – Reduce Number of Tapes

Tape Backups - Cons

• Media, interface, or drive errors

• Seldom validated

• Time to ship, travel and restore Hotsite

• Loss of data because backup only runs once per day

• Backups omitted in lieu of system access

• Lost tapes – Misplaced!

Servers - Un-Recoverable Data

Reasons for Lost Data

No Backup - Data lost

Incomplete Backup

Un-usable Backup

Other

0% 10% 20% 30% 40%

38% Incomplete

19% IBM Power

32% Windows

23% Oracle DB

17% Virtual Infrastructure

Tape Management

• Ensure tapes are labeled or cataloged with unique volume ID’s (BRMS/400, Robot Save)

• Prevent overwriting tapes with Active data

• Have at least 2 full system saves

• Audit tapes for data integrity

• Do NOT IGNORE tape drive problems

• PRTERRLOG *VOLSTAT



Do you have a SAVSYS on readable media?


How old is your recovery point?

Prevent a Disaster (Home & Away)

Is your security Flexible ?

• Who knows the QSECOFR password??

• Who knows the DST passwords ??

• Root, Admin, Routers – Back Doors ?

• Is your security model recoverable ?

• Physical Security

•

• 2574 document library objects saved.

• Starting save of list *LINK to devices TAP01.

43917 objects saved. 342 not saved.

Save of list *LINK completed with errors

• Starting save of media information at level *OBJ to device

• 18 objects saved from library QUSRBRM.

• Save of BRM media information at level *OBJ complete.

• DAILY *BKU 0070 *EXIT CALL PGM(BBSYSTEM/ENDDAYBU).

Control group DAILY type *BKU completed with errors.

Web Application Backup Log

No Backup Issues Around Here

• With a sign off, the process implies correctness

• Many backup solutions are partially broken

This equals 11 days per year with NO Backups

Reliable Backups

Backups are the backbone to recovery

• Most recoveries - The backups were not complete

• Excessive time is spent recreating parts of operating system

• Ensure compete recovery is possible from midweek or weekend failure

• System State is typically not complete

Missing System Level Components

Incorrect Attributes = *MISSING

Backup/Recovery Best Practices

• Examine policies for data recovery

• Examine policies for infrastructure recovery

• Off-site tape and Tape Retention Policies

• Have you audited your backups?

• Can you demonstrate a successful recovery from daily, weekly, and monthly backups?

Backup Horror Stories

• The Operating System Upgrade that Worked!

• Negligent Missing Objects – The Web App

• *Rewind – A Consulting Success Story

Is your data stored in a safe place ?

Recipe for Disaster

Ingredients:

One Average, everyday filing cabinet

All your important business documents

One Average business fire

Recipe for Disaster

Directions:

Place LTO backup media into cabinet.

Bake in fire at approximately 800F for 20min.

Let cool.

Open filing cabinet.

Recipe for Disaster

Result:

Your tapes and, for the most part, your

business is toast!

Offsite Strategy

DISASTER

STRIKES

10 AM

W T F S S M T W T F S S M T W T F S

FullBackup

S

OPERATIONS RECOVERY

POSSIBLEUNPROCESSED

DATA

PLANNED

OFFSITE

BACKUP

DAILY EXPOSURE

Incremental backup


Are your tapes accessible 7*24?

Are you ready for the worst case scenario?

Sandy: A Worthwhile Visit In History

• Lessons learned from a Regional Disaster

• Advance warning…Do NOT wait for the Disaster to take you down

Regional Disaster: Wake – Up Call

IT staff capabilities and availability

Modernization of current backup and recovery

solution

Data center, offsite storage, and hot site in same FEMA

region

Disasters Effect DR Supply Chain

Considerations:

• Is your regional Hot site solution provider experiencing same disaster?

• Understand SLA – Commitment by Suppliers

• Subscription Ratios

• Can your staff travel in a disaster?

How far is FAR enough?


FEMA Region Separation


HA – The Complete Solution

Is your HA truly High Availability?

Target and Source

– Same facility

Primary Facility Building Address:222 Cross My Fingers Drive


Issues with Side by Side

• Single point of failure

• Same power grid

• Same CO for communications

• No alterative in a Site Loss


Does Size Matter?

8 Cores

IBM Power 8

2 CoresIBM Power 6

H/A Common Findings

Mirrored System data integrity in serious

question

Data in-consistencies beyond application

missing or out of sync.

Little or NO documentation exists besides the

“Famous” Run book

Solution questioned by Management

Manage your HA High Availability is a living process

You must care and feed the solution

It is not just a product.

It’s a discipline

Qualified staff to manage and operate!

Consider the entire application service when protecting

…Verify what is in place works!!

Manage your HA

Ensure your HA solution provides

end-to-end replication in real time of current

production site - guaranteeing business

continuity and ROI


Common Misconceptions

It will never happen to me!

Have you seen how projects I am working on?

Business as usual after a security breach

We have nothing worth stealing in the first place!

Not in this years budget!

Murphy’s law: Incidents strike when, where, and because you

are not prepared


Lack of Security because…

• Time pressures!!

• Administrators wear too many hats.

Result:

• Poorly administered security schemes

• Too much authority

• Security software or products never installed or utilized


Is YOUR Security Watertight?

• Who has access to sensitive information?

• Who has access to key applications?

• Have violations occurred?

• Is your implementation strategy still being applied?


SECURITYPOLICY

SecurityStandard

IBMi

SecurityStandard

PC’sLAN Security

StandardOther

Systems

EmployeeGuidelines

Vendor&

ContractorGuidelines

Security Standards are an interpretation of the security

policy for IBM Power

Security Guidelines are an interpretation of the security

policy for user actions

Implementing Your Security Policy


Health Insurance Portability & Accountability Act ( HIPAA )

Sarbanes – Oxley ( SOX )

MA Data Breach Law 201 CMR 17

CA Security Breach Notification Act ( SB 1386 )

SSAE 16 ( previously SAS 70 )

SEC Rule 17a-4

International Traffic in Arms Regulation ( ITAR )

Federal Deposit Insurance Company (FDIC)

Payment Card Industry’s (PCI) Data Security Standards

Personal Information Protection & Electronic Documents Act ( PIPEDA)

Regulation Compliance The cost of non-compliance with these regulations can range from a few lost customers to millions of dollars in criminal and civil penalties.


A Not So Best Practice in Information Protection


Always Be Prepared

What is the first thing you would do if you suspected a breach?


We Have Some!!!


Implementing Your Security Policy

The value that i5OS provides is:

• Cost and effort to implement your security policy will be significantly less!

• Architecture, built-in function, ease-of-use means:

• Fewer explicit decisions/tasks required

• Easier to implement the choices (i.e. policy) you make


Biggest Security Exposure

Behind the firewall

• Disgruntled employees

• Accidental errors due to users having too much authority

No auditing

• No way to determine if there really is a problem


Limit the Tools available to a hacker

Only load software that is used

Common libraries QGPL, QUSRSYS

Source Editors - Secure DFU

Limit System access type Profiles

Actively monitor Special authorities

Exit programs need to be monitored

Set QSECOFR to STATUS (*DISABLED )

IBMi OS Security


Limit the Opportunities

Limit the opportunities for someone with ill intent to take advantage of your IBMi:

• Remove software no longer in use

• Secure source editors, e.g., DFU and DBU

• Disable and remove old profiles

• Limit the users with special authorities, especially *ALLOBJ

• Use object level security to protect data


• If you don’t audit, you have no knowledge of what happened.

• May need to audit to meet regulations - PIPEDA

• Minimum recommendation:

*SECURITY, *SAVRST, *AUTFAIL,

*DELETE, *CREATE, *SERVICE

Caution – Don’t audit too much!

No Auditing


Developers need copies to test:

• How much data do you provide

• The test Data is “real”

• Copies are often left unsecured on test servers

Unsecured Copies of Production Data


• Analyze use of Default passwords on your systems

• These are the first passwords a hacker will try

• Check Consultant & Suppliers passwords:

JDEINSTALL, JDEPROD, QPGMR, you’re Boss!!!

Change IBM & 3rd Party Default passwords

Default Passwords


Worst Passwords


Passwords are only as good as the policy that enforces their use.

That's why it's imperative that organizations employ a written password policy--and that they take steps to enforce it.

Password Management


• Rather than being cleaned up, profiles often accumulate, even though staff has left the company.

• Old profiles owning production Objects

Old User profiles


Systems have TCP/IP servers started even when they are not used.

• Check autostart attribute of servers

• These will start when STRTCP is run

• Check authority to STRTCPSVR

• This starts all TCP servers regardless of autostart

value

TCP/IP Applications


• Special authorities give users the ability to perform some functions

• *ALLOBJ special authority gives the user the ability to access ANY object on the system.

*ALLOBJ = DANGER !!!

• Don’t give users special authorities by default

Inappropriate Use of Special Authorities


Who have you empowered to query the data – Super users!!

Question – Do they need access to this information?

Don’t Forget Social Engineering


Most menu ‘Security’ designs assume:

• All access is through the application menu

• No users have command line access

• Query access is limited or denied

• There are no other applications that need to be interfaced with

• That the user is a member of the group that owns the objects

Reliance on Menu Security


ERP Risks – Control Tables

Sensitive ERP Master Files

Many master/control files contain sensitive company data

Address Book

Private sensitive information

Customer names & VISA details

Costs and Prices

Accidental updates & controls maintained.

Bill of Materials and Work files

Contain prices and costs and can be very sensitive.


Storing Data &

Monitoring Security

Compliance


Can you rebuild your system to satisfy the legal system?

Avoid piecing together or layering your recovery

• This is Not Disaster Recovery

You too will get audited - Count on That

• This auditor comes wearing a badge and carries a gun!

You can only rebuild what is on the backup tapes

• If they are expired then you are ……

Full system saves preferred as a baseline

• Consider keeping tapes longer then traditional Grandfather, Father, Son

Forensic Information – IBMi


Forensic Information – IBMiFor an i5/OS breach, here is the required information to begin the initial investigation:

• i5/OS audit journal receivers

• Other journal receivers (e.g., Vendor product receivers)

• History & Job logs

• Application Level - History and Audit logs

• Job Accounting History

• Network Access Logging

• Logical Security Reports

• Exit point software network access reports (ie: FTP, ODBC accesses)


Data Preservation – i5/OS

Disk Cleanup - Keeping the System Tidy

We have always been lectured to clean up the system - STOP!!

Keep all of your audit records complete

Special housekeeping saves performed to retain all audit records

• Keep these tapes aligned with Financial Records retention policies

• 3rd party software considerations


Encryption

Considerations


Tape and Data Encryption

• Choices for encrypting data on tape

• Encryption built into tape unit

• Encryption device between server and tape unit

• Encrypt sensitive data in DB fields

• Encrypt using 3rd party middleware for selected objects

Encryption Key Manager

(EKM) Server

System i

TS1120 Drives in

TS3500 Tape Library

Available

Sep 8/06

http://www-03.ibm.com/systems/i/hardware/520express/index.html

http://www-03.ibm.com/systems/i/hardware/520express/index.html


Tape Storage – Current and Archived


Successful Security Requires Teamwork

“The desire to secure your data is High.

Security is a partnership and ONLY successful

when everyone works together.”


UNDERSTAND AND ARE READY

UNDERSTAND BUT DON’T WANT DR PLAN

UNDERSTAND AND ARE NOT READY

DON’T UNDERSTAND. WHY BOTHER?

Security Preparedness

With that, I’m Out of Time…
