dns resilient

7/25/2019 DNS Resilient

1/263

Introduction to

Resilient TLD Operations

John Crain

Chris Evans


2/263

Welcome

Thank you! APTLD Leonid Todorov

ICANN Steve Conte & John Crain

2


3/263

Introductions

Name? Organization?

Duties?

3


4/263

Agenda

DNS Fundamentals DNS TLD Registries

Performance Monitoring

DNS Threats

Incident Management & Response

Resilient Enterprises

4


5/263

Administrivia

During the course: Ask questions when you have them no need to wait Your experiences are valuable please share them

Schedule can be flexiblejust ask!

Course Web Page http://files.delta-risk.net/projects/ICANN-Jakarta/

5


6/263

Questions before we begin?

6


7/263

DNS FUNDAMENTALS

7


8/263

What is the DNS?

The DNS is a distributed and hierarchical system of servers

which translates human readable names (identifiers) toaddresses

This allows people to remember simple names (e.g.

www.first.org instead of the actual server address,

213.129.76.19)

This is even more of a necessity with the advent of IPv6

Instead of 12 numbers for an address, there are 32 characters!

8

The Domain Name System (DNS) helps users to find their way aroundthe Internet

-- ICANN

128.63.2.53 2001:500:1::803F:235H.ROOT-SERVERS.NET

IPv4 IPv6


9/263

What is the DNS?

Every computer accessible on the Internet has aunique address, something like a telephone number,called an IP address

A unique name is given to each accessible computer,which maps to the IP address

The DNS provides resolution of these names to IPaddresses in a very transparent manner

9

The DNS is Often Compared to a Telephone

Book-- ICANN


10/263

What is the DNS?

Key Elements of the DNS Resolution

This is what the DNS does translates names to addresses

Distributed

There are literally millions of DNS servers all over the world. Thisadds resiliency to the system if one goes down there is usually

another that can function in its place.

Hierarchical

Responsibility for a DNS server for any given domain is delegated

to the respective owner of that domain, who, in turn, may

outsource those functions to a provider.

Consistent

Ideally, the same answer is provided regardless of the DNS server

being queried.10

icann.org

st.icann.org www.icann.org


11/263

A Brief History

March 1982 - Ken Harrenstien (et all) releases RFC 811governing use of a hostname server with a centrallymanaged file which contains names and addresses.

The ARPANetgrows People want local control

November 1983 Paul Mokapetris releases RFC 882and RFC 883 governing the concepts and operation ofthe DNS.

November 1987 Paul Mokapetris releases RFC 1033

and RFC 1034 which refine the DNS.The core pieces havent changed since (23 years)

11


12/263

DNS Operation

At a core level, the DNS is comprised ofQueries and Answers.

A user sends a query through a resolver to a

server. The server responds with an answerto the query.

12

User DNS Server

Query: What is www.icann.org?

Answer: 192.0.32.7


13/263

DNS Operation

A DNS packet is structured as such A packet can be up to 65535 bytes in length

It can have multiple queries and answers.

In theory most DNS servers will respond to only the first

query and ignore the rest.

13http://unixwiz.net/images/dns-query-packet.gif

Thats a lot of room for malicious

purposes


14/263

DNS Operation

14

User

ISP

Root

.org

icann

.org

pir

.org

first

.org


15/263

DNS Operation

15

A Tree view of the DNS

. (Root)

.COM

User

ISP

example.

com

www.example.com

.ORG

icann

.org

ftp.icann.org


16/263

DNS From Top To Bottom

The Root Servers

A collection of 13 serversaround the world, A M.

Actually, there are 202servers, all mirrors of the

same root zone data All are independently run by

root operators

16

www.root-servers.org

Highly redundant, resilient,

with diverse implementations Provide pointers to the

authoritative name servers


17/263


Example of what the rootservers provide:

BF. NS CASTOR.TELEGLOBE.NET.

BF. NS NAHOURI.ONATEL.BF.

BF. NS NS1.IRD.FR.

BG. NS SUNIC.SUNET.SE.

BG. NS NS-EXT.VIX.COM.BG. NS NS.REGISTER.BG.

BG. NS NS-BG.RIPE.NET

NAHOURI.ONATEL.BF. A 206.82.130.196

AUTH01.CONNECT.IE. A 213.228.227.50

OSI2.GUA.NET. A 205.161.188.3

NS.PA. A 168.77.8.2PARAU.OYSTER.NET.CK. A 202.65.32.128

PASCAL.NIC.PR. A 134.202.0.120

NS2.CUHK.EDU.HK. A 137.189.6.21

NS2.CUHK.EDU.HK. AAAA 2405:3000:3:60:0:0:0:21

17

These are the pointers to theauthoritative name servers

for a domain


18/263


Authoritative Servers Provides definitive

answers to queries for

resources in their

domains

ns.icann.org is an

authoritative server for

any queries to*.icann.org

18

These are usually delegated to the owner of the

domain, but sometimes outsourced. These can be run by individuals, businesses,

organizations, governments, really, anyone!


19/263

DNS From Top to Bottom

Authoritative servers publish Registration &Delegation information

When a user requests a new name to be added to the

DNS, its called a registration

When the registry assigns the name to the user, its calleda delegation they become the owners of the domain

and responsibility for any sub-domains falls to them.

Typically, their name, contact, and billing information is all

thats required but this varies greatly!

19

Theres no standard for registrations or

delegations so Ill pick whichever has the least

restrictive policies!


20/263


An example of a registration on a BIND DNS server We control the .foo domain

A registrant wants to register a host www.foo they choose the name

and provide an IP address

20

Registrations are not typically made at the top level domain delegations are

usually done here.

New Registration


21/263

If we query for our new registration, well seeit


21

New Registration


22/263

An example of a delegation on a BIND DNS server We control the .foo domain

A registrant wants to register a subdomain oof they

choose the name, and provide authoritative nameserver

information


22

New Delegation


23/263

Any queries for the new subdomain should failbecause the authoritative nameserver doesnt exist

(aka a lame delegation)


23

Queries to the

Authoritative

Server


24/263


Caching / Recursive Servers

These servers interface with

authoritative servers to answer

queries.

Answers are cached to speed

up responses to users and reducetraffic to root servers

24

Usually run by ISPs for customers,

but should be configured for

customer use only!

There are many of these

servers on the net

Open resolvers answer queries from

anyone on the net.

See http://dns.measurement-

factory.com/surveys/openresolver

s.html for more information!


25/263


Users & Stub Resolvers

Not just people, these could also

be applications & servers

A stub resolver is a simple

utility which forms queries and

asks a recursive server for theanswer.

Remember, users also can be

registrants the people who

register domain names

25

An operating system will typically

provide a stub resolver for its users


26/263


The DNS Should be Considered Critical Infrastructure,and as such, it has a lot of resilience built-in to thesystem: Primary and Secondary Servers a zone may have several

servers which respond to queries

Hidden Masters the master server cant be querieddirectly it merely replicates data to the Slave servers

Anycasted Servers BGP routing forward queries toclosest server

How Does All of This Stay in Sync? Zone Transfers (aka AXFR) secondary requests update

from primary

26


27/263

DNS TLD REGISTRIES

27


28/263

Registry Structures

Registries Come in All Shapes and Sizes Structure, Technical Implementation, Goals &

Policies

Models Define How the Registry Operates 2R vs. 3R

Thick vs. Thin

2 vs. 3 Levels

28


29/263

Registry Structures

Registry 2R Model Registry - Registrant

Registry provides direct registrations to consumers

Registrants request domains directly through registry

29

Registry

RegistrantRegistrant

Simple model

Doesnt scale without

lots of work from registry

Direct customer

relationships


30/263

Registry Structures

Registry 2R Model Registry - Registrant

Registry provides direct registrations to consumers

Registrants request domains directly through registry or

optionally through a direct reseller

30

Simple model

Direct customer

relationships

Offloads some customer

interface work fromregistry to reseller

Registry

Registrant

Registrant

Reseller


31/263

Registry Structures

Registry 3R Model Registry Registrar Registrant

Registrars interface between customer and registry

31

More stakeholders ->

more policies / business

models

Less work for registry(automation / EPP)

Focus on DNS services

Registry

Registrant

Reseller

Registrar

Registrant

Registrar


32/263

Registry Structures

Registries can define WHERE their customeradministrative data resides (WHOIS)

Thick the registry stores all customer data for

registrations regardless of _WHERE_ it originates (direct,

registrar) Thin Registrars manage all customer data for their

registrants and serve this information via WHOIS

32

TLD

WHOIS

TLD

WHOIS

Registrar

Registrar

Th ick Th in

Registrar


33/263

Registry Structures

Registries define how subdomains are permittedeither at the top or second level.

Top level delegations fall immediately under the

top level

E.g. www.google.jo

Second level delegations fall under a generic

domain, under the top level

E.g. www.moict.gov.jo

Registries can pick one or the other, or do both!

33
http://www.google.jo/http://www.moict.gov.jo/http://www.moict.gov.jo/http://www.google.jo/


34/263

Notional ccTLD Architecture

Combining Robustness & Resiliency

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

34


35/263


Registrant Requests Assignment, Updates, Removal

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

35


36/263


Authentication for Registrant Requests

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

36


37/263


Authorization for Internal

Registrar Changes

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

37


38/263


Entire Registry Offsite Backup

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

38


39/263


Registry Publishes and Maintains Assignments

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

39


40/263


Alternate Registry Server and Database

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

40


41/263


Country Localized DNS Servers

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

41


42/263


Country Localized User

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

42


43/263


Firewall

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

43


44/263


Primary Global DNS

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

44


45/263


Primary External Gateway

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

45


46/263


Secondary Global DNS Server

Anycasting with Geographic Separation

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

46


47/263


Secondary External Gateway

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

47


48/263


International User

Registrant

NS2

NS1

User

NS4

NS3Internal

Internal

External

External

InternationalUser

48


49/263

Policies

Policies govern how registries operate, from internal

workings to dealing with customers.

Policies can be derived from operational necessities,

business goals, or best practices in use by other registries

Policies vary by registry and vary nearly as widely as theregistries themselves.

Its important for registries to define policies to respond and

operate in a consistent manner

Frequently, there is no right or wrong policy what mattersis the registrys definition of that policy.

49

The wrong policy is NO policy


50/263

Policy

Registrant Requirements This policy defines the requirements to register a

domain with the registry; specifically addresses issueslike residency or ownership

Things that may restrict registrations to only localor national entities: Desire to limit competition from foreign entities

Desire for nationalism

Things that may encourage registration fromforeign entities: Access to a larger pool of customers

50


51/263

Policy

Dispute Resolution A controversial subject; one that is used infrequently, but

must be defined before its needed!

Disputes may arise between potential registrants for adomain name each claiming the right to register it,

- or -

- Disputes may arise between current registrant a one thatdesires ownership of the domain

Policy should define the notion of ownership, the

process of determining ownership, and process forarbitrating a claim of ownership First come / first served = ownership

International trademark or copyright = ownership

51


52/263

Policy

Registration Process This policy defines how the registry accepts requests

for registrations, validates, and publishes them

Usually defined by operational necessity Larger registries that process hundreds of registrations per

day will use automated processes as much as possible Smaller registries may use manual processes

Policy should cover registrant data verification

Automated, multi-factor, out-of-band, etc Desire to detect and restrict malicious registrations?

How should the registry define malicious?

52

Tough

Question!


53/263

Policy

Information Release (WHOIS) This policy defines how the registry handles privacy

information and what information it publicly publishes

The traditional approach is to publish administrative andtechnical point of contact information for each domain

BUT, some registries / registrars offer a anonymous registrationservice which publishes empty or generic information

This may be governed by local, national, or organizationalprivacy policies.

This policy should cover what information can bepublished, to who & where, how the policy isconveyed to registrants, and what to release to lawenforcement or government agencies.

53


54/263

Policy

Pricing / Funding Model This policy defines how much domain registrations, their

renewal, transfer or other action cost.

Additional or value-added services may also be defined

This policy should address costs to registrants, registrarsand resellers accordingly and is likely defined by theregistrys business model. Registries set their own prices, from free to $$$$

Some registries are funded by their governments, some arenon-profits, some are for-profit corporations.

Competition with gTLDs (e.g. .COM) may drive pricing decisions Prices may be different for local and foreign registrants, for

resellers, and for registrars

54

l


55/263

Policy

Permissible Registrations This policy defines what domain names may be

registered

Factors affecting permissible registrations: Religion & Culture some names may be offensive

Technical some characters are disallowed by theRFC, or some may not be allowed by the registrysystem

Operational International Domain Names may notbe supported

Business some names may be restricted based oninternational trademarks or intellectual property

55

li


56/263

Policy

Takedown

A very controversial topic, this defines how the registrymay temporarily or permanently deactivate a registration

More importantly, under what circumstances it may do so.

Policy should account for:

Business concerns / loss of revenue

Upset customers?

Malicious use (what is malicious?)

Legality of denying freedom of speech

Fallout of not following a government / law enforcementorder

56

h i l l


57/263

A Hypothetical Example

Lets take the simple example of a registration made for the purpose

of conducting a phishing attack. The domain name closely resembles that used by an international

bank

The registrant provides fake name and address for registration

The registrant uses the domain to propagate a phishing attack,

soliciting bank customers for their usernames and passwords.

Whos responsible and whos accountable? The malicious registrant? How do you find him?

The registry? They didnt validate the registrant but they get 5000

registrations a day how can they? The bank? They didnt educate their customers on phishing attacks

The bank user?

57

T k A


58/263

Take Aways

The DNS operationally is simple; organizationally, its a much

different story there are many (widely varying) stakeholders

involved

Policies define how registries operate, but are frequently ill-

defined and untested

Unlike the traditional network security world, which has

CERTs and NOCs focusing on operating systems and

applications, the DNS has no equivalent structure for focusing

on security

58


59/263

PERFORMANCE MONITORING

59

N t k P f M t i


60/263

Planning performance management

Metrics

Network

Systems

Services

Definitions

Adapted from IROC Hervey Allen & Phil Regnauld - NSRC

Network Performance Metrics

Pl i


61/263

What's the intention?Baselining, Troubleshooting, Planning growth

Defend yourself from accusations -it's the network!

Who is the information for?Administration, NOC, customers

How to structure and present the information

Reach: Can I measure everything? Impact on devices (measurements and measuring)

Balance between amount of information and time to get

it

Planning

M t i


62/263

Metrics

Network performance metrics Channel capacity, nominal & effective

Channel utilization

Delay and jitter Packet loss and errors

62

M t i


63/263

Metrics

System performance metrics Availability

Memory, CPU Utilization, load, I/O wait, etc.

Service performance metrics Wait time / Delay

Availability

How can I justify maintaining the service? Who is using it? How often?

Economic value? Other value?

63

Common Network Performance


64/263

Measurements

Relative to traffic: Bits per second

Packets per second

Unicast vs. non-unicast packets

Errors

Dropped packets

Flows per second

Round trip time (RTT)

Jitter (variation between packet RTT)

64

N i l Ch l C it


65/263

Nominal Channel Capacity

The maximum number of bits that can betransmitted for a unit of time (eg: bits per second)

Depends on:

Bandwidth of the physical medium

Cable Electromagnetic waves

Processing capacity for each transmission element

Efficiency of algorithms in use to access medium

Channel encoding and compression

65

Effective Channel Capacity


66/263

Effective Channel Capacity

Always a fraction of the nominal channelcapacity

Dependent on:

Additional overhead of protocols in each layer Device limitations on both ends

Flow control algorithm efficiency, etc.

For example: TCP

66

Channel Utilization


67/263

Channel Utilization

What fraction of the nominal channelcapacity is actually in use

Important!

Future planning What utilization growth rate am I seeing?

For when should I plan on buying additional capacity?

Where should I invest for my updates?

Problem resolution Where are my bottlenecks, etc.

67

95th Percentile


68/263

95th Percentile

The smallest value that is larger than 95% of the

values in a given sample

This means that 95% of the time the channel

utilization is equal to or less than this value

Or rather, the peaks are discarded from consideration

Why is this important in networks?

Gives you an idea of the standard, sustained channel

utilization.

ISPs use this measure to bill customers with larger

connections.

68

95th Percentile


69/263

95th Percentile

Example of traffic above 95% of normal

69

Bits per second vs Packets p s


70/263

Bits per second vs Packets p.s.

70

End to end Delay


71/263

End-to-end Delay

The time required to transmit a packet alongits entire path

Created by an application, handed over to the OS,

passed to a network card (NIC), encoded,

transmitted over a physical medium (copper,fibre, air), received by an intermediate device

(switch, router), analyzed, retransmitted over

another medium, etc.

The most common measurement uses ping for

total round-trip-time (RTT).

71

Historical Measurement of Delay


72/263

Historical Measurement of Delay

Example of RTT measurements over 30 hours

72

Jitter


73/263

Jitter

From Smokeping

73

Local Analysis


74/263

Local Analysis

As we know...

Before we blame the network, let's verify

whether the problem is ours.

What can go wrong locally?

Hardware problems

Excessive load (CPU, memory, I/O)

What's considered 'normal'?

Use analysis tools frequently Become familiar with the normal state and values

for your machine.

It is essential to maintain history

SNMP agents and databases74

Linux Performance Analysis


75/263

Linux Performance Analysis

Three main categories:

Processes

Processes that are executing (running)

Processes that are waiting (sleeping)

waiting their turn

blocked

Memory

Real

Virtual

I/O (Input/Output) Storage

Network

75

Key Indicators


76/263

Key Indicators

Insufficient CPU

Number of processes waiting to execute is always high

High CPU utilization (load avg.)

Insufficient memory

Very little free memory

Lots of swap activity (swap in, swap out)

Slow I/O

Lots of blocked processes

High number of block transfers

76

Local Analysis


77/263

Local Analysis

Luckily, in Unix there are dozens of useful

tools that give us lots of useful information

about our machine

Some of the more well-known include:

vmstat - tcpdump top - wireshark (ethereal)

lsof - iptraf

netstat - iperf

77

vmstat


78/263

vmstat

Show periodic summary information aboutprocesses, memory, pagin, I/O, CPU state, etc

vmstat # vmstat 2procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----r b swpd free buff cache si so bi bo in cs us sy id wa2 0 209648 25552 571332 2804876 0 0 3 4 3 3 15 11 73 02 0 209648 24680 571332 2804900 0 0 0 444 273 79356 16 16 68 01 0 209648 25216 571336 2804904 0 0 6 1234 439 46735 16 10 74 01 0 209648 25212 571336 2804904 0 0 0 22 159 100282 17 21 62 0

2 0 209648 25196 571348 2804912 0 0 0 500 270 82455 14 18 68 01 0 209648 25192 571348 2804912 0 0 0 272 243 77480 16 15 69 02 0 209648 25880 571360 2804916 0 0 0 444 255 83619 16 14 69 02 0 209648 25872 571360 2804920 0 0 0 178 220 90521 16 18 66 0

78

top


79/263

top

Basic performance tool for Unix/Linuxenvironments

Periodically show a list of system performance

statistics:

CPU use

RAM and SWAP memory usage

Load average (cpu utilization)

Information by process

79

top cont


80/263

top cont.

Information by process (most relevantcolumns shown):

PID: Process ID

USER: user running (owner) of the process

%CPU: Percentage of CPU utilization by the

process since the last sample

%MEM: Percentage of physical memory (RAM)

used by the process TIME: Total CPU time used by the process since it

was started

80

netstat


81/263

netstat

Show us information about: Network connections

Routing tables

Interface (NIC) statistics

Multicast group members

81


82/263

netstat


83/263

netstat

Examples:# netstat -n --tcp -cActive Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 272 ::ffff:192.188.51.40:22 ::ffff:128.223.60.27:60968 ESTABLISHEDtcp 0 0 ::ffff:192.188.51.40:22 ::ffff:128.223.60.27:53219 ESTABLISHED

# netstat -lnp --tcpActive Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0.0.0.0:199 0.0.0.0:* LISTEN 11645/snmpdtcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1997/mysqld

# netstat -icKernel Interface tableIface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1500 0 2155901 0 0 0 339116 0 0 0 BMRUlo 16436 0 18200 0 0 0 18200 0 0 0 LRU

eth0 1500 0 2155905 0 0 0 339117 0 0 0 BMRUlo 16436 0 18200 0 0 0 18200 0 0 0 LRUeth0 1500 0 2155907 0 0 0 339120 0 0 0 BMRUlo 16436 0 18200 0 0 0 18200 0 0 0 LRUeth0 1500 0 2155910 0 0 0 339122 0 0 0 BMRUlo 16436 0 18200 0 0 0 18200 0 0 0 LRUeth0 1500 0 2155913 0 0 0 339124 0 0 0 BMRU

83

netstat cont.


84/263

netstat cont.

Examples:# netstat tcp listening --programActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 *:5001 *:* LISTEN 13598/iperftcp 0 0 localhost:mysql *:* LISTEN 5586/mysqld

tcp 0 0 *:www *:* LISTEN 7246/apache2

tcp 0 0 t60-2.local:domain *:* LISTEN 5378/named

tcp 0 0 t60-2.local:domain *:* LISTEN 5378/namedtcp 0 0 t60-2.local:domain *:* LISTEN 5378/named

tcp 0 0 localhost:domain *:* LISTEN 5378/named

tcp 0 0 localhost:ipp *:* LISTEN 5522/cupsdtcp 0 0 localhost:smtp *:* LISTEN 6772/exim4

tcp 0 0 localhost:953 *:* LISTEN 5378/named

tcp 0 0 *:https *:* LISTEN 7246/apache2tcp6 0 0 [::]:ftp [::]:* LISTEN 7185/proftpd

tcp6 0 0 [::]:domain [::]:* LISTEN 5378/named

tcp6 0 0 [::]:ssh [::]:* LISTEN 5427/sshd

tcp6 0 0 [::]:3000 [::]:* LISTEN 17644/ntoptcp6 0 0 ip6-localhost:953 [::]:* LISTEN 5378/namedtcp6 0 0 [::]:3005 [::]:* LISTEN 17644/ntop

84

netstat cont.


85/263

$ sudo netstat -atup

Active Internet connections (servers and established) (if run as root PID/Program name isincluded)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 *:35586 *:* LISTEN 2540/ekpd

tcp 0 0 localhost:mysql *:* LISTEN 2776/mysqldtcp 0 0 *:www *:* LISTEN 14743/apache2

tcp 0 0 d229-231.uoregon:domain *:* LISTEN 2616/named

tcp 0 0 *:ftp *:* LISTEN 3408/vsftpdtcp 0 0 localhost:domain *:* LISTEN 2616/named

tcp 0 0 *:ssh *:* LISTEN 2675/sshd

tcp 0 0 localhost:ipp *:* LISTEN 3853/cupsd

tcp 0 0 localhost:smtp *:* LISTEN 3225/exim4tcp 0 0 localhost:953 *:* LISTEN 2616/named

tcp 0 0 *:https *:* LISTEN 14743/apache2

tcp6 0 0 [::]:domain [::]:* LISTEN 2616/namedtcp6 0 0 [::]:ssh [::]:* LISTEN 2675/sshd

tcp6 0 0 ip6-localhost:953 [::]:* LISTEN 2616/named

udp 0 0 *:50842 *:* 3828/avahi-daemon:udp 0 0 localhost:snmp *:* 3368/snmpd

udp 0 0 d229-231.uoregon:domain *:* 2616/namedudp 0 0 localhost:domain *:* 2616/named

udp 0 0 *:bootpc *:* 13237/dhclientudp 0 0 *:mdns *:* 3828/avahi-daemon:

udp 0 0 d229-231.uoregon.ed:ntp *:* 3555/ntpd

udp 0 0 localhost:ntp *:* 3555/ntpdudp 0 0 *:ntp *:* 3555/ntpd

udp6 0 0 [::]:domain [::]:* 2616/named

udp6 0 0 fe80::213:2ff:fe1f::ntp [::]:* 3555/ntpdudp6 0 0 ip6-localhost:ntp [::]:* 3555/ntpd

udp6 0 0 [::]:ntp [::]:* 3555/ntpd

netstat cont.

85

lsof (LiSt of Open Files)


86/263

lsof (LiSt of Open Files)

lsof is particularly useful because in Unixeverything is a file: unix sockets, ip sockets,

directories, etc.

Allows you to associate open files by:

-p: PID (Process ID)

-i : A network address (protocol:port)

-u: A user

86

lsof


87/263

lsof

Example: First, using netstat -lntcp determine that port6010 is open and waiting for a connection

(LISTEN)

# netstat -ln --tcpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN

87

lsof


88/263

lsof

Determine what process has the port (6010) open

and what other resources are being used:# lsof -i tcp:6010COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEsshd 10301 root 6u IPv4 53603 TCP localhost.localdomain:x11-ssh-offset (LISTEN)sshd 10301 root 7u IPv6 53604 TCP [::1]:x11-ssh-offset (LISTEN)

# lsof -p 10301COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEsshd 10301 root cwd DIR 8,2 4096 2 /sshd 10301 root rtd DIR 8,2 4096 2 /sshd 10301 root txt REG 8,2 379720 1422643 /usr/sbin/sshdsshd 10301 root mem REG 8,2 32724 1437533 /usr/lib/libwrap.so.0.7.6sshd 10301 root mem REG 8,2 15088 3080329 /lib/libutil-2.4.sosshd 10301 root mem REG 8,2 75632 1414093 /usr/lib/libz.so.1.2.3

sshd 10301 root mem REG 8,2 96040 3080209 /lib/libnsl-2.4.sosshd 10301 root mem REG 8,2 100208 1414578 /usr/lib/libgssapi_krb5.so.2.2sshd 10301 root mem REG 8,2 11684 1414405 /usr/lib/libkrb5support.so.0.0sshd 10301 root mem REG 8,2 10368 3080358 /lib/libsetrans.so.0sshd 10301 root mem REG 8,2 7972 3080231 /lib/libcom_err.so.2.1sshd 10301 root mem REG 8,2 30140 1420233 /usr/lib/libcrack.so.2.8.0sshd 10301 root mem REG 8,2 11168 3080399 /lib/security/pam_succeed_if.so...

88

lsof cont.


89/263

lsof cont.

What network services am I running?# lsof -iCOMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

firefox 4429 hervey 50u IPv4 1875852 TCP 192.168.179.139:56890-

>128.223.60.21:www (ESTABLISHEDnamed 5378 bind 20u IPv6 13264 TCP *:domain (LISTEN)

named 5378 bind 21u IPv4 13267 TCP localhost:domain (LISTEN)

sshd 5427 root 3u IPv6 13302 TCP *:ssh (LISTEN)

cupsd 5522 root 3u IPv4 1983466 TCP localhost:ipp (LISTEN)mysqld 5586 mysql 10u IPv4 13548 TCP localhost:mysql (LISTEN)

snmpd 6477 snmp 8u IPv4 14633 UDP localhost:snmp

exim4 6772 Debian-exim 3u IPv4 14675 TCP localhost:smtp (LISTEN)ntpd 6859 ntp 16u IPv4 14743 UDP *:ntp

ntpd 6859 ntp 17u IPv6 14744 UDP *:ntp

ntpd 6859 ntp 18u IPv6 14746 UDP [fe80::250:56ff:fec0:8]:ntpntpd 6859 ntp 19u IPv6 14747 UDP ip6-localhost:ntp

proftpd 7185 proftpd 1u IPv6 15718 TCP *:ftp (LISTEN)

apache2 7246 www-data 3u IPv4 15915 TCP *:www (LISTEN)

apache2 7246 www-data 4u IPv4 15917 TCP *:https (LISTEN)...

iperf 13598 root 3u IPv4 1996053 TCP *:5001 (LISTEN)

apache2 27088 www-data 3u IPv4 15915 TCP *:www (LISTEN)apache2 27088 www-data 4u IPv4 15917 TCP *:https (LISTEN)

89

iptraf


90/263

iptraf

Many measurable statistics and functions By protocol/port

By packet size

Generates logs

Utilizes DNS to translate addresses

Advantages

Simplicity

Menu-based (uses curses)

Flexible configuration

90

iptraf


91/263

pt a

You can run it periodically in the background(-B)

It allows you, for example, to run as a cron job to

periodically analyze logs.

Generate alarms

Save in a data base

Has a great name... Interactive Colorful IP LAN

Monitor

etc...

Example: iptraf -i eth1

91

iptrafi eth0


92/263

p

Sample iptrafoutput from the above command:

92

iperf


93/263

p

To measure network throughput between two

points

iperf has two modes, server and

client

Easy to use Great to help determine optimal TCP parameters

TCP window size (socket buffer)

MTU maximum segment size See man iperf for more

93

iperf


94/263

p

Using UDP you can generate packet loss and jitter

reports

You can run multiple parallel sessions using threads

Supports IPv6

94

iperf parameters


95/263

Usage: iperf [-s|-c host] [options]iperf [-h|--help] [-v|--version]

Client/Server:-f, --format [kmKM] format to report: Kbits, Mbits, KBytes, MBytes-i, --interval # seconds between periodic bandwidth reports-l, --len #[KM] length of buffer to read or write (default 8 KB)-m, --print_mss print TCP maximum segment size (MTU - TCP/IP header)-p, --port # server port to listen on/connect to-u, --udp use UDP rather than TCP-w, --window #[KM] TCP window size (socket buffer size)-B, --bind bind to , an interface or multicast address-C, --compatibility for use with older versions does not sent extra msgs-M, --mss # set TCP maximum segment size (MTU - 40 bytes)

-N, --nodelay set TCP no delay, disabling Nagle's Algorithm-V, --IPv6Version Set the domain to IPv6

Server specific:-s, --server run in server mode-U, --single_udp run in single threaded UDP mode-D, --daemon run the server as a daemon

Client specific:-b, --bandwidth #[KM] for UDP, bandwidth to send at in bits/sec

(default 1 Mbit/sec, implies -u)-c, --client run in client mode, connecting to -d, --dualtest Do a bidirectional test simultaneously-n, --num #[KM] number of bytes to transmit (instead of -t)-r, --tradeoff Do a bidirectional test individually-t, --time # time in seconds to transmit for (default 10 secs)-F, --fileinput input the data to be transmitted from a file-I, --stdin input the data to be transmitted from stdin-L, --listenport # port to recieve bidirectional tests back on-P, --parallel # number of parallel client threads to run-T, --ttl # time-to-live, for multicast (default 1)

p p

95

iperf - TCP


96/263

$ iperf -s------------------------------------------------------------Server listening on TCP port 5001TCP window size: 85.3 KByte (default)------------------------------------------------------------[ 4] local 128.223.157.19 port 5001 connected with 201.249.107.39 port 39601[ 4] 0.0-11.9 sec 608 KBytes 419 Kbits/sec

------------------------------------------------------------

# iperf -c nsrc.org

------------------------------------------------------------

Client connecting to nsrc.org, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.1.170 port 39601 connected with 128.223.157.19 port 5001[ 3] 0.0-10.3 sec 608 KBytes 485 Kbits/sec

p

96

iperf - UDP


97/263

# iperf -c host1 -u -b100M------------------------------------------------------------Client connecting to nsdb, UDP port 5001Sending 1470 byte datagramsUDP buffer size: 106 KByte (default)------------------------------------------------------------[ 3] local 128.223.60.27 port 39606 connected with 128.223.250.135 port 5001[ 3] 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec[ 3] Sent 81377 datagrams[ 3] Server Report:

[ 3] 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%)

$ iperf -s -u -i 1------------------------------------------------------------Server listening on UDP port 5001Receiving 1470 byte datagramsUDP buffer size: 108 KByte (default)------------------------------------------------------------[ 3] local 128.223.250.135 port 5001 connected with 128.223.60.27 port 39606[ 3] 0.0- 1.0 sec 11.4 MBytes 95.4 Mbits/sec 0.184 ms 0/ 8112 (0%)[ 3] 1.0- 2.0 sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms 0/ 8141 (0%)[ 3] 2.0- 3.0 sec 11.4 MBytes 95.6 Mbits/sec 0.182 ms 0/ 8133 (0%)...[ 3] 8.0- 9.0 sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms 0/ 8139 (0%)[ 3] 9.0-10.0 sec 11.4 MBytes 95.7 Mbits/sec 0.180 ms 0/ 8137 (0%)[ 3] 0.0-10.0 sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%)

p

97

Bibliography


98/263

g p y

Monitoring Virtual Memory with vmstat

http://www.linuxjournal.com/article/8178 How to use TCPDump

http://www.erg.abdn.ac.uk/users/alastair/tcpdump.html

linux command tcpdump examplehttp://smartproteam.com/linux-tutorials/linux-command-tcpdump/

simple usage of tcpdumphttp://linux.byexamples.com/archives/283/simple-usage-of-tcpdump/

TCPDUMP Command man page with exampleshttp://www.cyberciti.biz/howto/question/man/tcpdump-man-page-with-examples.php

TCPDump Tutorial

http://inst.eecs.berkeley.edu/~ee122/fa06/projects/tcpdump-6up.pdf

98


99/263

DNS THREATS

99

Cyber Threats


100/263

y

Open Question What Cyber Threats

Concern You?

Disruption of DNS Resolution Services

Falsified Registrant Information

Someone Stealing Your Customer Data

Hackers using Your Registry for BotNet Control

100

100

Cyber Threat Categorization


101/263

Foothold

Reconnaissance & Enumeration

Privilege Escalation

Corruption

Disruption Data Exfiltration

Persistence

Malicious Use

101

Foothold Recon & Enum Priv Esc.

Corruption

Persistence

Data Exfiltration

Disruption

Time & Effort

Concurrency

Malicious Use

Some Attacks May Involve Multiple

Categories

Foothold


102/263

Initial attempts to access and establish a remote connection

into the network Direct

SQL Injection, Operating System Exploits

Indirect

Phishing Email with Malware Attachments Website Hosted Malware Installations

Motivations

Gain Access for Future Attacks

102

This is usually an

enabling attack

Foothold Attacks


103/263

These attacks will target your servers and

workstations anything that is accessible over the

network

103

NS

Accessible Email, Web,

Remote Administration, etc

NS

Foothold Attacks - Phishing


104/263

104

fraudwatchinternational.com

http://news.softpedia.com/news/PDF-Based-Targeted-Attack-

Against-Military-Contractors-Spotted-212139.shtml

Foothold Attacks MitigatingStrategies


105/263

Strategies

Awareness & Education Phishing Exercises

Intrusion Detection Systems

Critical Application Code Reviews

Externally Visible System & Application Patching Access Control White listing

Conduct Regular Vulnerability Scans

Finds common holes which an attacker can exploit

Others?

105

Reconnaissance & Enumeration


106/263

The act of scanning a network to determine its layout,

hosts, services, users, and other information whichmay be useful in a cyber attack Network Mapping

Port Scanning

Motivations Political : DNS Administrators May List Sensitive

Information in Zone Comments Financial : Can be Used to Find Addresses That are NOT

Currently Registered for Domain Squatting Malicious: Can be Used to Obtain Information for Use in

Future Attacks

106

Reconnaissance & EnumerationAttacks


107/263

Attacks

These attacks will target your externally visible

servers, and potentially your internal network hosts

as well

107

NS

NS

Business & Operations

Networks

Reconnaissance & EnumerationExample


108/263

Example

Zone Transfer

Used to Retrieve All Information From a DNS Server Normally Used by Secondary to Sync with Primary

108

C:\nslookup

Default Server: ns1.example.net

Address: 127.0.0.1

> set type=any> ls -d example.net [dns1.example.net]

example.net. SOA dns01.example.net test.dns02.example.net. (1

3600 600 86400 3600)

example.net. NS dns01.example.net


example.net. MX 10 email.example.net

testpc TXT "payment processing computer"

testpc A 10.10.10.2

adminspc TXT Administrators PC"

adminspc A 10.10.10.3

test CNAME adminspc.test.net

Other DNS Servers



109/263

Example

Zone Transfer


109

C:\nslookup


Address: 127.0.0.1



3600 600 86400 3600)





testpc A 10.10.10.2




Email Server



110/263

Example

Zone Transfer


110

C:\nslookup


Address: 127.0.0.1



3600 600 86400 3600)





testpc A 10.10.10.2




Potentially Sensitive Computers

Identified by Comments

Reconnaissance & EnumerationMitigating Strategies


111/263

Mitigating Strategies

Access Control

White Listing Restrict Zone Transfers to Only Trusted Secondary Servers

Validate The Trusted Paths Used Transaction Signatures (TSIG) for Zone Transfers

Network Monitoring Know Whos Looking at You

Internal & External

Block Repeated Offenders

But dont worry about intermittent port scans or spend alot of time here IPs can change on a whim!

Information Control Limit the available information (ports, banners, zone file)

111

Privilege Escalation


112/263

Obtaining credentials, beyond what is

normally available, for the purpose of

accessing systems or information

Username / Password Cracking

Social Engineering

Exploiting Trust Relationships

Motivations

Gain Access to Information or for Future Attacks112

Privilege Escalation Attacks


113/263

Privilege Escalation attacks will target systems or

relationships that contain something of value.

Value could be information, or perhaps, increased

access to a system to perform another attack

113

NS

NS


Networks

Privilege Escalation Examples


114/263

Password cracking can target your Operating Systems, Web

Applications, even your Wireless Networks Password guessing and dictionary attacks are forms of password

cracking short passwords and words found in_any_ language

dictionary are susceptible.

Trust Relationships with Service Providers Can be Abused

How do you validate EPP requests from your registrars?

Do you have business functions that rely on external providers (e.g.

credit card processors) that can be abused?

114

Privilege Escalation MitigatingStrategies


115/263

Strategies

Use Strong / Complex Passwords

Operating System, Web Application, WiFi,_anything_ thatuses a password

Use a passphrase, not a single dictionary word

Network Monitoring

Track failed login attempts Track usage of trusted paths (e.g. EPP requests)

Intrusion Detection Systems Internal and External keep signatures up to date

Anti-virus / Anti-spyware

Others?

115

Corruption


116/263

Modifying information or content on a system or application

Website Defacement

Cache Poisoning

Name Server Redirection

Motivations Political : Make Public Statements on Behalf of Your ccTLD or the

Government

Financial: Data Ransoming Attacker May Try to Sell You the Key toUnlock Your Own Data

Malicious: Hackers Want to Make a Name for Themselves, DenyService, or Redirect Users

Accidental: Employees make mistakes!

116

011101001101

Corruption Attacks


117/263

Corruption Attacks will Target Your Data Stores

File Servers, Name Servers

May Also Target Recursive Name Servers

117

NS

NS


Networks

Insider too!

Corruption Attacks


118/263

118

http://www.rankmyhack.com/

http://www.thehackernews.com/2011/08/rankmyhack-got-hacked-by-haxor.html

So many defacements

http://www.zone-h.org/archive/special=1

Corruption Example


119/263

Name Server Registration Redirection

Recall CheckFree.com!

Non-Authoritative Spoofing

Partido Colorado political statement made by

one organization against another during an

election

119

Attacker Modifies Upstream DNS Records

www.partidocolorado.org. 3600 IN A

66.249.01.121

Corruption Example


120/263

Partido Colorado the ISP (Copaco) modified its own

recursive name servers to redirect users looking forpartidocolorado.org

120

Attacker Modified Upstream DNS Records

www.partidocolorado.org. 3600 IN A 66.249.01.121

201.217.51.114

Corruption Mitigating Strategies


121/263

External Monitoring

Your Own Systems, and Recursive Systems But this may not be tractable for you

Internal Monitoring Compare to Known Good (read-only!) Copy

Implies you are making backups on a regular basis

Two-Person Authorization for Critical Processes

Detection Becomes Critical Monitor User / Registrant Communications Monitor Traffic Usage for Sites You Control

A Significant and Sudden Drop in Traffic Could be a Problem Authenticate All External Requests for Updates

Require two factor authentication for automated registration updates

Out of band communication like fax, in-person, etc Communicate Issues with Upstream Server Operators

Others?

121

Disruption


122/263

Denying, degrading, or otherwise limiting the

availability of services provided by a system or

application

Packet Floods

Deleting Operating System Files

Motivations Malicious: Denial of Service

Political: Denial of Service (Cyber Riots)

Badge of Honor: DoS the Root DNS Servers

Financial: Demonstrate Power of BotNet122

Disruption Attacks


123/263

These attacks target your external servers and/or

the links which service them Regular Queries

How Many Queries per Second Can Your Servers Handle?

How Much Bandwidth Do You Have?

Targeting Your Single Points of Failure

123

NS

NS

Disruption Attacks


124/263

These attacks target your external servers and/or the links

which service them; single points of failure

124

Limited Scale

http://www.darkgovernment.com/news/tsa-data-analyst-planned-

cyber-attack/

http://www.ibtimes.com/articles/131421/20110406/anonymous-launches-

ddos-attack-on-sony.htm

Low Orbit Ion Cannon =Cyber Rioting

Disruption Mitigating Strategies


125/263

Understand Where Your Single Failure Points

Are Add Duplicate Servers and Capacity Where One

Exists

Geographic Separation / Anycasting Country Localized and Global Servers

Ensure Automated Data Replication

Close Contact with Service Providers

Others?

125

Data Exfiltration


126/263

Copying data, that is not publicly accessible, from a

network for malicious purposes

Smash & Grab

Corporate Espionage

Motivations Malicious: Taint Reputation of a Registry

Financial: Data Ransoming, Identity Theft

126

Data Exfiltration Attacks


127/263

These Attacks Target Your Internal Servers and

Information - Stealing Anything of Value

This is especially easy for an Insider someone who is

authorized to be on your network, and conducts malicious

activity

127

NS

NS


Networks

Data Exfiltration Attacks


128/263

128

http://www.privacyrights.org/data-breach

http://datalossdb.org/

These are just the reported onesNo corporate espionage here!

Data Exfiltration Example


129/263

A Data Thief Targets a ccTLD

Gains unauthorized access to customer database

Copies the database of customer names, email addresses,phone numbers, addresses and credit card details

Sells this information to others on the black market for $5

per identity

129

A Purely Hypothetical Example

Data Exfiltration Mitigating Strategies


130/263

Prevent Foothold and Privilege Escalation Attacks

Keep Continuous Backups

And Test Them!

Enforce Strict Access Controls to Sensitive Data

Does engineering need access to customer data?

Consciously Decide to Keep Information on Servers

Do you really need copies of customer data or just a

single source of information?

Network Monitoring

Create a Baseline of Your Normal Traffic

Continuously Compare Current Traffic to Baseline

130

Persistence


131/263

Establishing an unauthorized, constant, presence on the

network and thwarting administrator removal attempts Root Kits

Monitoring Administrator Actions

Motivations

Malicious: Desire to Stay Within Your Network to Gain AdditionalAccess or Perform Additional Attacks

131

011101001101

Persistence Attacks


132/263

These Attacks May Target Any Host in Your Network,

from Servers to Workstations

Your External Service Providers are Not Immune Either!

132

NS

NS


Networks

Persistence Example


133/263

An Attacker Compromises a Host Through

Phishing

Immediately installs a rootkit to hide evidence of

a successful compromise

Rootkit hides files, the interactive shell process,and the network ports which the attacker uses to

access the compromised host

133

Persistence Attacks


134/263

134

http://www.theregister.co.uk/2011/06/29/tdss_alureon_advances/http://www.computerworlduk.com/news/security/3288274/microsoft-warns-rootkit-

infection-requires-windows-reinstall/?olo=rss

Hides files, processes, network ports,

provide backdoors, advanced C2 & P2P

Persistence Mitigating Strategies


135/263

Anti-Virus and Anti-Spyware Scans

Signatures Must Be Kept Updated

Network Traffic Monitoring

Create a Traffic Baseline

Compare Current Traffic to that Baseline Proxy Server Logs, IDS, Traffic Logs, etc

Intrusion Detection System

Signatures for common root kits, unexpected encoded

traffic, unexpected encrypted traffic

Conduct Regular Vulnerability Scans

Finds common holes which an attacker can exploit, finds

common rootkit ports135

Malicious Use


136/263

Using the DNS to propagate malware or conduct

attacks in a malicious manner, yet consistent withthe DNS protocols

BotNet Command & Control

Amplification Attacks

Motivations

Malicious: Spreading malware through servers hiding

behind fictitious DNS registrations

Financial: For Hire BotNets use DNS for command and

control

136

Malicious Use Attacks


137/263

These Attacks Do Not Necessarily Target Your

External DNS Servers Rather, They Use YourServers to Conduct an Attack Elsewhere

137

NS

NS

Victim

Malicious Use Example Conficker


138/263

Conficker - the Conficker worm appeared in late 2008, with

most of the attention starting in Jan/Feb of 2009. The worm used pseudo-randomly generated domains from several top

level domains (ccTLDs included) as its command and control points.

The worm would contact servers on these random domains for

instructions.

138

Malicious Use Example Angler


139/263

The Angler exploit kit has been active since 2014.

Continuously updated with new 0-day exploits

Domain shadowing - uses subdomains of legit domain

names for command and control

Rapidly shifts exploit pages on subdomains

139

http://blogs.cisco.com/security/talos/angler-domain-shadowing

Malicious Use Mitigating Strategies


140/263

Define a Policy for Malicious Use

How would you respond to requests to pre-register adomain name which might be used for maliciouspurposes?

How do you validate registrant information?

What are your domain takedown policy and procedures?

Network Monitoring Know what your expected baseline is monitor for

anomalies

Limit Size of Query Responses

Be a good steward of the Internet

Others?

140

Putting it All Together


141/263

Phishing email530 emails, 57 clicked link

Link to exploit site

0-day exploit

Malware installationData exfiltration round 1

Initial Cleanup

Dormant malware goes liveData exfiltration round 2

141

http://www.wired.com/threatlevel/2011/04/oak-ridge-lab-hack/


142/263

INCIDENT MANAGEMENT &

RESPONSE

14

2

Incident Management Introduction


143/263

Incident Management

Introduction

143

Definitions and Acronyms


144/263

CSIRT Computer Security Incident Response Team

Responsible for cyber incident response processes CERT Computer Emergency Response Team

Also called Computer Emergency Readiness Team

Often synonymous with the CSIRT/CIRT

Can contain more functions and ops than a CSIRT 2nd and 3rd tier analysis

Research

Coordination

Often associated with a country, industry, or community CIRT Computer Incident Response Team

Similar to a CERT, often more company focused

144

Definitions and Acronyms


145/263

SOC Security Operations Center

Primarily intrusion prevention and monitoringfunctions in the security domain

NOC Network Operations Center Monitoring health and function of the network

NOSC Network Operations and Security Center Combination of SOC and NOC

IRT Incident Response Team Functional team found within a CERT, SOC, or NOSC

Can also be a stand-alone team within a company

IRP Incident Response Plan

145

Incident Response Vs. IncidentHandling


146/263

Whats the difference?

Incident Response

The technical, hands-on, what happened? work

Incident Handling

The project management, lead-the-team work

Or, are they synonyms?

Heated online battles by industry giants

The NIST documentation says they aresynonymous

You decide for yourself

146

Events and Incidents


147/263

Event vs. Incident

Event: Any occurrence of any type

Incident: Damage, Degradation, or Persistent Intent to do so

Depends on your policy and thresholds

Events may be:

Unsuccessful access attempts

Poor internal security activities

Recon attempts (no impact)

Anything that is currently investigated

According NIST, Incidents include:

Unauthorized Access Denial of service

Presence of malicious logic

Improper Usage147

Introduction to Incident Response


148/263

The many different phases of Incident

Response

Preparation Identification Containment Eradication RecoveryLessonsLearned

Detect

Respond

Prevent

(Law Enforcement I.R.)Preparation

Preservation

Duplication

Analysis

Reporting

148

The Phases of Incident Response


149/263

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

Hot-wash Damage Assessment Applying Lessons

System RestoreRepeat Attack

MonitoringIT/User Interaction

Tool Usage Rootkits Polymorphic Code APT

Deployment System Management Forensics

Sensors Logs Traffic Flow Correlation Declaration

Policy ManagementSupportInfo

Sharing

Law/

MediaMetrics Exercise

Phases Sub-Steps & Cons iderat ions

149

Preparation


150/263

Preparation is key. If you do

not properly prepare, yourincident response team will

experience failures when stress

levels are high

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

150

Detection & Analysis


151/263

When prevention methods fail,

detection is your only way tofind out that something is

happening. You will detect

many events, but they can turninto incidents quickly

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

151

Containment

h k b d


152/263

The attacks must be stopped

quickly before too muchdamage occurs. This is where

forensics and defensive tactics

come into play. Time is of theessence.

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

152

Eradication

d d


153/263

In order to prevent immediate

re-infection, the systems mustbe purged of any artifacts left

by the attacker. This is where

they will be cleaned up.

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

153

Recovery

N i i b i h


154/263

Now its time to bring the

systems back online and putthem into production (if you

took them offline). If proper

steps were taken, the systemshould be infection free.

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

154

Lessons Learned

Th i f


155/263

The importance of

documenting what occurredcannot be understated. There

are undoubtedly process

improvements and policychanges that need to be made.

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

155

Anatomy of Incident Response


156/263

In order to prevent immediate re-

infection, the systems must be purged

of any artifacts left by the attacker.

This is where they will be cleaned up.

Now its time to bring the systems

back online and put them into

production (if you took them offline). Ifproper steps were taken, the system

should be infection free.

The importance of documenting what

occurred cannot be understated.

There are undoubtedly process

improvements and policy changes

that need to be made.

The attacks must be stopped quickly

before too much damage occurs. This

is where forensics and defensivetactics may come into play. Time is of

the essence.

When prevention methods fail,

detection is your only way to find out

that something is happening. Mostly,

you will detect events, but they can

turn into incidents quickly.

Preparation is key. Failure to properly

prepare your teams for incident

handling will result in failures whenstress levels are high.

Preparation

Detection

& Analysis

Containment

Eradication

Recovery

Lessons

Learned

156

Where Does an IRT Fit In?

Notional CIRT


157/263

Incident

Response Team

Full-timeDetection &

Analysis

Forensics

Intelligence AdvancedTraffic

Monitoring

Legal, Management, Policy

&Planning, Training,Exercise, Evaluation

Notional CIRTYou may have

some, all, or noneof these!

157


158/263

INCIDENT RESPONSE

Case Study

15

8

The Backstory


159/263

You run a ccTLD registry

You have a typical setup consisting of primary

and secondary DNS servers and a registry service

running on a web server.

For simplicity assume the web and primary DNS areon the same server hosted in your office / data center

You receive an urgent call from a registrant

that their primary name servers have been re-

delegated without their permission to a

server hosted by 00webhost.com

159

Incident Response - Preparation

What questions would you ask of


160/263

What questions would you ask of

registrants to understand the issue? What technical measures would you

take on your servers to determine

what happened?

A side discussion what

logging/monitoring would you need

to have in place to make sure you

could answer those questions?

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

160

Incident Response - Detection &Analysis

For this case study assume your alerting


161/263

For this case study, assume your alerting

tools didnt produce anything

Your customer alerted you!

For discussion:

What would your tools tell you?

Do you have visibility into:

Login attempts (web, ssh, all other services)

Current active connections to your servers

Changes to data (zone file, high value domains)

Changes to system (new users, cron jobs,installed programs, file system)

Centralized logging and/or log files

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

161

Incident Response - Tools


162/263

Live Forensics Toolset Brian Moran

https://www.brimorlabs.com/tools/

Excellent cheat sheets from Lenny Zeltser

https://zeltser.com/cheat-sheets/

162

https://www.brimorlabs.com/tools/https://www.brimorlabs.com/tools/https://www.brimorlabs.com/tools/https://zeltser.com/cheat-sheets/https://zeltser.com/cheat-sheets/https://zeltser.com/cheat-sheets/https://www.brimorlabs.com/tools/


163/263

You find the following as part of your analysis. What stands

out? cat /etc/passwd

cat /etc/group

root@kali:~# cat /etc/passwd

root:x:0:0:root:/root:/bin/bash

daemon:x:1:1:daemon:/usr/sbin:/bin/sh

dns:x:124:124::dns:/bin/bash

test:x:125:125:test:/bin/bashdnsadmin:x:126:126:dnsadmin:/bin/bash

root@kali:~# cat /etc/group

root:x:0:daemon:x:1:

admin:x:111:dnsadmin,test,dns

stunnel4:x:129:

sambashare:x:130:

163



164/263


out? cat /var/log/auth.log

But wait cat /var/log/auth.log.1 dont forget log rotations

Sep 2 06:25:47 kali CRON[4240]: pam_unix(cron:session): session closed for user root

Sep 2 06:28:56 kali sshd[4126]: Received disconnect from ::1: 11: disconnected by user

Sep 2 06:28:56 kali sshd[4126]: pam_unix(sshd:session): session closed for user root

Sep 2 05:47:26 kali dbus[2947]: [system] Rejected send message, 2 matched rules; type="method_call",

sender=":1.27" (uid=123 pid=3632 comm="/usr/lib/gdm3/gdm-simple-greeter ") Sep 2 05:47:49 kali gdm3][3645]:

pam_unix(gdm3:auth): authentication failure; logname= uid=0 euid=0 tty=:0 ruser= rhost= user=root

Sep 2 05:47:57 kali gdm3][3652]: pam_unix(gdm3:auth): authentication failure; logname= uid=0 euid=0 tty=:0

ruser= rhost= user=root

Sep 2 05:48:03 kali gdm3][3654]: pam_unix(gdm3:session): session opened for user root by (uid=0)

Sep 2 05:48:03 kali gdm3][3654]: pam_ck_connector(gdm3:session): nox11 mode, ignoring PAM_TTY :0

Sep 2 05:48:03 kali gdm-welcome][3235]: pam_unix(gdm-welcome:session): session closed for user Debian-gdm

Sep 2 05:48:03 kali polkitd(authority=local): Unregistered Authentication Agent for unix-

session:/org/freedesktop/ConsoleKit/Session1 (system bus name :1.26, object path

/org/gnome/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

Sep 2 06:28:20 kali gnome-keyring-daemon[3658]: unsupported key algorithm in certificate: 1.2.840.10045.2.1

Sep 2 06:18:21 kali sshd[4126]: Accepted password for test from port 55019 ssh2

Sep 2 06:18:21 kali sshd[4126]: pam_unix(sshd:session): session opened for user test by (uid=125)

Sep 2 06:25:01 kali CRON[4240]: pam_unix(cron:session): session opened for user root by (uid=0)

Sep 2 06:32:31 kali CRON[4086]: pam_unix(cron:session): session opened for user test by (uid=125)

Sep 2 06:42:22 kali CRON[4086]: pam_unix(cron:session): session closed for user test164


165/263



166/263


out? cat /var/log/syslog

Sep 3 17:03:07 NS1 named[965]: listening on IPv4 interface eth0, 192.168.201.105#53

Sep 3 17:17:01 NS1 CRON[2734]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Sep 3 18:17:01 NS1 CRON[2978]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)Sep 3 19:17:01 NS1 CRON[3222]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Sep 3 20:17:01 NS1 CRON[3599]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)Sep 3 20:58:52 NS1 init: tty1 main process ended, respawning

Sep 3 21:01:54 NS1 crontab[3971]: (test) BEGIN EDIT (test)Sep 3 21:03:14 NS1 crontab[3971]: (test) REPLACE (test)

Sep 3 21:03:14 NS1 crontab[3971]: (test) END EDIT (test)

Sep 3 21:03:20 NS1 crontab[4002]: (test) LIST (test)Sep 3 21:04:13 NS1 crontab[4019]: (root) BEGIN EDIT (root)

Sep 3 21:04:17 NS1 crontab[4019]: (root) END EDIT (root)

Sep 3 21:05:47 NS1 crontab[4059]: (root) BEGIN EDIT (root)

Sep 3 21:06:03 NS1 crontab[4059]: (root) REPLACE (root)Sep 3 21:06:03 NS1 crontab[4059]: (root) END EDIT (root)

166


f d h f ll f l h d


167/263


out? cat /etc/crontab also crontabe -lroot@kali:~# cat /etc/crontab

# /etc/crontab: system-wide crontab

# Unlike any other crontab you don't have to run the `crontab'# command to install the new version when you edit this file

# and files in /etc/cron.d. These files also have username fields,

# that none of the other crontabs do.

SHELL=/bin/sh

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user command

17 * * * * root cd / && run-parts --report /etc/cron.hourly

25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report/etc/cron.daily )

47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report

/etc/cron.weekly )52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report

/etc/cron.monthly )

55 0 0,12 * * * test nc l p 8080 e /bin/bash#

167


168/263


Y fi d h f ll i f l i Wh d


169/263


out? cd /home && findcmin -10 (enter whatever time you need

here)

find /homenewermt 2015-09-03 00:00:00 ! newermt

2015-09-04 00:00:00

root@NS1:/# cd /home && find -cmin -10

./test

./test/tmp

./test/tmp/hspferdata_test

./test/tmp/hspferdata_test/rnicks.e

./test/tmp/hspferdata_test/rversions.e

./test/tmp/hspferdata_test/rinsult.e

./test/tmp/hspferdata_test/16148

./test/tmp/hspferdata_test/fa.out

./test/tmp/hspferdata_test/cron.d

169


Y fi d th f ll i t f l i Wh t t d


170/263


out? Login as test user then, history

What does this indicate?

170

test@NS1:~$ history

1 history


171/263

Incident Response - Containment

What would you do to fix this and


172/263

What would you do to fix this and

prevent the attacker fromcontinuing an attack?

Lots of options pros and const

Shut down the system entirely

Disable root and/or test accounts

Reset passwords on all accounts

Kill active ssh session

Remove crontab entry

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

172

Incident Response - Eradication

What would you do to clean up the


173/263

What would you do to clean up the

mess? Search for and remove recently created

files?

Delete any user accounts not

recognized Lock down SSH to be only accessible by

user account and/or IP address

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

173


174/263

Incident Response - Lessons Learned

For discussion:


175/263

For discussion:

How would you get notified ofthis attack without relying on the

customer?

Did your response actions work?

Did you have enough staff to

handle the response?

Preparation

Detection &Analysis

Containment

Eradication

Recovery

LessonsLearned

175


176/263

Resiliency

protection and sustainabilityof


177/263

p ycritical services, associatedbusiness processes, and assets

177

Service

Process Process

Assets

PEOPLE INFORMATION FACILITIESTECHNOLOGY

Adapted from SEI | CERT Resiliency Management Model


178/263

Resiliency Activities

Resilienc is being able to react to stress


179/263

Resiliency is being able to react to stress

without breaking

Many different ways of being resilient

Several frameworks exist to help guide your

resiliency activities

179

NIST SP 800-53 SANS / CSC Top 20

ASD Mitigation Strategies ISO 27002

And many others!
https://web.nvd.nist.gov/view/800-53/homehttps://www.sans.org/critical-security-controls/http://http/www.asd.gov.au/publications/protect/top_4_mitigations.htmhttp://www.27000.org/iso-27002.htmhttp://www.27000.org/iso-27002.htmhttp://http/www.asd.gov.au/publications/protect/top_4_mitigations.htmhttps://www.sans.org/critical-security-controls/https://web.nvd.nist.gov/view/800-53/home


180/263

Resiliency Frameworks

CSC (SANS) Top 20


181/263

CSC (SANS) Top 20

20 things you can do today, includes guidance, recommended tools,implementation ideas

Consensus of community experts

ASD Mitigation Strategies 35 things you can do to increase resiliency

Based on things that would have stopped 85% of the attacks that ASD

responded to

Use these if you have limited resources and need to know

where to get started

181

ASD Examples


182/263

Top 4 Application whitelisting

Patch application

Patch OS

Restrict administrative

privileges

Others Application hardening

Host-Based IDS

Disable local administrator

accounts

Network segmentation

Multi-factor authentication

Application firewalls

Web filtering

Strong passwords Network traffic capture

Event logging

182


183/263


184/263

Resiliency in Practice Redundancy

Having multiple copies of something


185/263

Having multiple copies of something

DNS servers, service provider links, payment

systems, data backups, others?

Doesnt need to be running at the same time

But it does need to be available on short notice

Your recovery time objectives will say how much

short notice is acceptable!

185


186/263

Secure Operations Framework

A Process for Conducting Secure Operations


187/263

A Process for Conducting Secure Operations

Within Your Registry

187

Monitor Detect

Respond

RecoverAnalyze

Baseline

Secure Operations Framework

Viewed as a Timeline:


188/263

Viewed as a Timeline:

188

Baseline Monitor Detect Analyze RecoverRespond

Do This

Before This

Your Detection

Time May Vary

ATTACKQuick!

Stop Effect

Prevent


189/263

Establishing a Baseline

Architecture Baseline


190/263


Hosts, Services, Ports, Connections, Addresses

This is frequently a network architecture

diagram, but could be a text document, or

part of your network monitoring solution

190




191/263


Document your host configurations, operating

systems, applications, versions, etc

Ideally you want something you can reference

quickly during attack analysis e.g. Can you quickly answer the question:

191

Is my web server vulnerable to thelatest Apache vulnerability?


Architecture Baseline Tools


192/263

Architecture Baseline Tools

Visio

Paper / Pencil!

NAGIOS, OpenNMS, HP Openview, etc

Lots of tools available the hardest part is

actually doing it!

192


Traffic Baseline


193/263

Traffic Baseline

What kind, how much, source / destination, andusual time

This is more difficult to capture but is critical

to determining if something is expected ornot

193


External Traffic Baseline


194/263

How much traffic (packets per second, megabits persecond, etc) is normal on your external links?

What kind of traffic to external servers (e.g. do younormally have SSH connections to your DNS servers?)

Internal Traffic Baseline Which hosts or applications communicate?

What protocols do they use?

How much traffic do they produce?

When do they do it?

194


Traffic Baseline


195/263

Traffic Baseline

Again, ideally, you will have this informationreadily available during the analysis phase

For example, can you easily answer the question:

195

Is a SSH connection to my DNS server at

0330 on a Saturday normal?


Traffic Baseline Tools


196/263

NetFlow Wireshark

Tcpdump

Iperf, dnsperf

Again, lots of tools available, the hard part is notonly capturing the baseline traffic, but putting it

into a format you can reference Graphing and statistical analysis tools can help here!

SmokePing, NetFlow Analyzer

196


People Baseline


197/263

p

How do your customers interact with your network? Web application for processing registration?

Do you typically see people login at 0330?

How do your administrators interact with yournetwork?

What hosts do they connect to, what protocols do they use,when do they do it?

How do your local users interact with your network? What hosts are they _supposed_ to be on, what hosts do

they communicate with, when?

How do your external users interact with yournetwork?

What servers do they connect to, what protocols?

197


People Baseline


198/263

People Baseline

Again, ideally, you will have this informationreadily available during the analysis phase

For example, can yo

dns resilient

Documents