cisco firewall

Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.SEC-340 3143_05_2001_c1_X.scr

2© 2003, Cisco Systems, Inc. All rights reserved.SEC-30208243_06_2003_X2

Troubleshooting FirewallsSession SEC-3020



Agenda

• Understanding the Concepts

• PIX and FWSM Troubleshooting Tools

• PIX and FWSM Common Issues

• IOS Firewall Concepts

• IOS Firewall Troubleshooting Tools

• IOS Firewall Common Issues


Agenda









How the PIX Processes a Packet

PublicPrivate

Entering PIX Leaving PIXPacket Processed

Against ASA

Adaptive Security Algorithm

Randomize Sequence NumbersXlate & Connection ObjectsStateful InspectionSecurity LevelsOther rules


Security = 40

Security = 0Corp Security = 30

Security = 100

Security Levels

• Security Levels Specify the Trust between interfaces

• Access from a higher security level (100) to a lower (40)security requires a nat/global or static translation

• Access from a lower security level (0) to a higher security level (30) requires a static and an access-list/conduit

• Traffic between same security levels (DMZ = 30) and (DMZ1 = 30)

Security = 30 Eng

Outside

Inside



Agenda








PIX and FWSM Troubleshooting Tools

• Syslog• Debug• Packet Capture (used only on appliance)• Tracebacks (6.3)• ACL Logging (6.3)• Show Commands• Output Interpreter• PDM (3.0)



What Is Syslog?

• Message sent by the PIX to report activity to and thru the PIX

• Appear on your console, buffer and syslog server

Private Public

%PIX-6-308001: PIX console enable password incorrect for num tries (from IP_addr)

Explanation: This is a PIX Firewall management message. This message is logged after the num number of times a user incorrectly types the password to enter privileged mode. The maximum is three attempts.

Action: The privileged mode password is not necessarily the same as the password for Telnet access to the PIX Firewall. Verify the password and try again.

Example of Syslog Message


Configuring and Using Syslog

pixfirewall (config) # Logging trap 7

pixfirewall (config) # Logging host inside 171.68.9.137

pixfirewall (config) # Logging console 7

pixfirewall (config) # Logging buffered 7

pixfirewall (config) # Logging on

Log level for messages to

Syslog server

Host = IP address of Syslog server

Log level for serial console port

Log level for internal buffer

Turn Logging on



Reading a Syslog Message

Message ID: See PIX Syslog documentation for specific explanations

Severity Level: Range 1-7, the lower the number, the more severe the condition.

%PIX-6-308001: PIX console enable password incorrect for 3tries (from 171.68.89.147)

This message was generated by a …


Reading a Syslog Message

Source IP Address

Destination IP Address

Type/Code of Message

AccessControl List

Message ID Protocol

%PIX-4-106023: Deny icmp src outside:171.68.88.1dst inside:171.68.89.147(type 3, code 1) by access-group "outside_access_in"



What Are Modifiable Syslog Levels

• Modifiable Syslog LevelsAllows one to move any syslog message to any syslog level.

• Example:You want to see what web sites users are going to by logging message PIX-5-304001, which by default is at level 5 (Notification)

%PIX-5-304001: [email protected] Accessed URL 198.133.219.25:/tac

However, you really don’t want to see all the other syslog messages at level 5 or level 4. Instead, you want to log level 3 and below, along with the Accessed URL message.

[no] logging message <syslog_id> level <level>

Levels

0 – emergency

1 – alert

2 – critical

3 – errors

4 – warnings

5 – notifications

6 – informational

7 - debugging


How to Create Modifiable Syslog Levels

[no] logging message <syslog_id> level <level>

Solution:

• Lower syslog message 304001 to level 3 (Error)

PIX(config)# logging message 304001 level 3- or -

PIX(config)# logging message 304001 level error

• Now our syslog looks as follows:%PIX-3-304001: [email protected] Accessed URL 198.133.219.25:/tac

• To restore the default syslog level: PIX(config)# no logging message 304001 level error - or –

PIX(config)# logging message 304001 level 5



Clear Modifiable Syslog Levels

• To restore all of the currently changed syslogs back to their default levels:

PIX(config)# clear logging level • To view all messages with modified levels:

PIX# show logging levelsyslog 304001: default-level notification, current-level error (enabled)

• To view all disabled messages:

PIX# show logging disabledno logging message 106023

clear logging levelshow logging [{message [<syslog_id>|all]} | level | disabled]

Question: If a user only wants to display a couple syslogs, what is the easiest way to accomplish this?


View Modifiable Syslog Levels

•To view a particular message: PIX# show logging message 304001 syslog 304001: default-level notifications, current-level errors (enabled)

•To view all messages with modified levels, and all disabled messages:

PIX# show logging message syslog 302010: default-level informational (disabled)syslog 304001: default-level notifications, current-level errors (enabled)

To view all syslog messages and their current and default level, and their status:

PIX# show logging message all

show logging [{message [<syslog_id>|all]} | level | disabled]



Verify Syslog

PIX515c# show logging

Syslog logging: enabledTimestamp logging: disabledStandby logging: disabledConsole logging: level debugging, 404 messages loggedMonitor logging: disabledBuffer logging: level debugging, 5 messages loggedTrap logging: level debugging, facility 20, 404 messages logged

Logging to inside 171.68.9.137

History logging: disabled

302002: Teardown TCP connection 1 faddr 172.16.171.235/23 gaddr 172.16.171.234/1025 laddr 172.16.171.3/22548 duration 0:00:23 bytes 274 (TCP FINs)

Logging Turned On

Different Output Locations

Example of a syslog message


Syslog Levels in PIX

9 (265)Debugging787 (256)Informational619 (169)Notifications538 (140)Warnings449 (102)Errors318 (53 possible)Critical235Alerts10Emergencies0# of MessagesDescriptionLog Level




• Syslog• Debug • Packet Capture (used only on appliance)• Show Commands• Tracebacks (6.3)• ACL Logging (6.3)• Output Interpreter• PDM (3.0)


An Introduction to Debug

• Detail level analysis to help troubleshoot protocols operating with and through the PIX Firewall.

• Viewed while using a console or telnet session to the PIX

• Excellent tool totest basic IP connectivity and complex issues

Debug are CPU intensive

What are they ?

Where can I view them ?

Why use them?

Any Issues to Consider ?

Debug Packet<option>Debug ICMP

Trace



Debug ICMP Trace

Private Network

Internet

http://www.xyz.com

11 User able to access Internet? No

22 Can access private network? Yes

33 Any Syslog messages? No

44 Test IP connectivity at this time

12: Outbound ICMP echo request (len 32 id 2 seq 33538) 10.10.10.1 > 63.1.1.5 > 200.1.1.113: Inbound ICMP echo reply (len 32 id 2 seq 33538) 200.1.1.1 > 63.1.1.5 > 10.10.10.1

Example of Debug ICMP Trace to test successful ip connectivity

55 pixfirewall (config) # debug icmp trace


Notes on Debugging ICMP Trace

Bob

Inside

DMZ

Outside Internet

11

• User can only ping interface directly connected to it11

22

• ICMP has to be explicitly permitted thru the PIX Firewall to test IP connectivity

22



Is user able to access private network ? Yes

11 User able to access Internet ? No

22

33 Anything showing up on Syslog ? No

Debug Packet <option>

http://www.xyz.com

Private Network

Internet

44 pixfirewall (config) # debug packet <option>

• This Debug will tell the user if the packet is received on the PIX interface

• Give the user detailed session info, for eg IP address, ports and flags

• Should control the display of this debug by limiting it to IP address and ports

• Can be effectively used in conjunction with Syslog to troubleshoot complex issues


Example of Debug Packet <option>

--------- PACKET ----------- IP --10.10.10.1 ==> 200.1.1.1.

ver = 0x4 hlen = 0x5 tos = 0xc0 tlen = 0x2cid = 0x0 flags = 0x0 frag off=0x0ttl = 0xff proto=0x6 chksum = 0x4c4c

-- TCP --source port = 0x2af9 dest port = 0x50ack

seq = 0x471c8bbback = 0x0hlen = 0x6 window = 0x1020checksum = 0x1ef9 urg = 0x0

tcp options: 0x2 0x4 0x2 0x18 -- DATA --

0000002c: 00 00 cc | ...--------- END OF PACKET ---------

TCP

Web traffic on port 80



Disabling Debug Commands

In 6.3 you can now turn off all debugs globally by issuing "no debug all" and "undebug all” or “un all” for short.

PIX(config)# show debugdebug icmp tracedebug sip

PIX(config)# un allPIX(config)# show debugPIX(config)#

undebug allundebug all



• Syslog• Debug • Packet Capture (used only on appliance)• Tracebacks (6.3)• ACL Logging (6.3)• Show Commands• Output Interpreter• PDM (3.0)



What Is Packet Capture

• The primary goal of this feature is to produce a standalone tool on the PIX that captures packets and is useful for network fault isolation.

• [ethernet-type <type>] to capture Layer 2 packets such as Arp and PPPoEpackets

• Output can be either to Console, tftp file, browser screen using SSL and save it to pcap file using: copy capture:<capture-name> tftp://<location>/<pathname> [pcap]https://<PIX’s ip address>/capture/<capture-name>[/pcap]

• Pcap or decoded ASCII format is supported

Outside Network

Inside Network

Tftp server


Example of a Capture Command

access-list 100 permit tcp host 200.10.10.10 host 195.1.1.1 eq 80access-list 100 permit tcp host 195.1.1.1 eq 80 host 200.10.10.10capture outside1 access-list 100 interface outside

Capturing Traffic on Outside Interface

Viewing Captured Trafficcopy capture:outside1 tftp://10/1/1/10 pcapORhttps://<PIX_IP>/capture/outside1/pcap

No. Time Source Destination Protocol Info

15 148.701751 10.1.1.1 192.168.1.1 TCP 4511>http[SYN] Seq=27007623614 Ack=0

16 148.704086 192.168.1.1 10.1.1.1 TCP http>4511[SYN,ACK] Seq=979356760..

17 148.705398 10.1.1.1 192.168.1.1 TCP 4511>http[ACK] Seq=2707623615………

18 148.701751 10.1.1.1 192.168.1.1 HTTP GET /HTTP/1.1



Packet Capture Output Format

ICMP packet:

<HH:MM:SS.ms> [ether-hdr] <ip-source> > <ip-destination>: icmp: <icmp-type> <icmp-code> [checksum-failure]

UDP packet:

<HH:MM:SS.ms> [ether-hdr] <src-addr>.<src-port> <dest-addr>.<dst-port>:[checksum-info] udp <payload-len>

TCP packet:

<HH:MM:SS.ms> [ether-hdr] <src-addr>.<src-port> <dest-addr>.<dst-port>:

<tcp-flags> [header-check] [checksum-info] <sequence-number> <ack-number> <tcp-window> <urgent-info> <tcp-options>

Other IP packets:

<HH:MM:SS.ms> [ether-hdr] <src-addr> <dest-addr>: <ip-protocol> <ip-length>

ARP packets: <HH:MM:SS.ms> [ether-hdr] <arp-type> <arp-info>

802.1Q: <HH:MM:SS.ms> [ether-hdr] <VLAN-info> <encap-ether-packet>

Other packets: <HH:MM:SS.ms> [ether-hdr] <hex-dump>


Capture Command FAQ’s

303030SEC-3205203_05_2002_c1 30

• You can have one capture running per interface. This allows you to capture packets as they enter one interface, and again as they go out another interface

• Currently, the default capture buffer is a 512kb and once the buffer is full, the PIX stops capturing packets

• The buffer is saved in RAM so before you create the buffer make sure you have enough RAM available.

• PIX Version 6.3 expands the capture to allow a circular buffer

• Once you have captured the packets needed, copy them off via TFTP or HTTPS






Introduction to Traceback

• What is a TracebackIn case of any unusual packet that PIX can’t process. PIX will sometimes dumps the memory contents to the console and reboots.

• What causes a TracebackIn case of any unusual packet that PIX can’t process; PIX will sometimes dumps the memory contents to the console and reboots.

• How does one capture itBeginning with 6.3 crash information can be saved to the flash and can be retrieved.

• How do I analyze itYou don’t. Forward it to TAC and we will analyze it for you ☺



Some Important Traceback Commands

[show|clear] crashinfo :Displays and clears crash info from flash

crashinfo save [enable|disable] : Enable or disable of saving of crash into the flash.(default behavior of the PIX)

show crashinfo save :Shows whether crash info saving is enable or disabled.

no crashinfo save disable : Disables automatic saving of crash into flash


What Does a Traceback Look Like

Thread Name: pptp_mgmt (Old pc 0x8024d489 ebp 0x83f2d8c4)

Traceback:0: 8024da321: 8024c7b42: 8024d45e3: 800030294: 00000000

vector 0x0000000e (page fault)edi 0x840ea7a0esi 0xffffffffebp 0x83f2d844esp 0x83f2d80cebx 0x840e9cb8edx 0x00000009ecx 0x00000024eax 0x94b5d1dc

error code 0x00000000eip 0x8024c88f

.... more of stack dump ....Cisco PIX Firewall Version 6.2(2)114Cisco PIX Device Manager Version 2.0(2)

Snip of Traceback






ACL Logging

In PIX 6.2 and below, customers could issue a “show access-list” to see the hit count against an ACL line. Syslogs messages would record every packet that was denied, and each connection that was created, but it was very difficult to correlate or aggregate this information. Also, the deny logs could be overwhelming.

access-list outside-acl permit ip host 1.1.1.1 any (hitcnt=10) access-list outside-acl deny ip host 2.2.2.2 any (hitcnt=3)

106023: Deny icmp src outside:2.2.2.2 dst inside:7.0.0.2 (type 0, code 0) by access-group "outside-acl"



Syslogs by ACL in 6.3

• Add a log option to the end of an ACE. Also provide a mechanism to rate-limit the syslogs generated from the log option.

• The logging is only applicable to ACLs that are configured by the 'access-group' command. As such, only 'through-traffic' is subject to logging.

access-list <acl_id> permit|deny ... [log [disable|default] | [<level>] [interval <secs>]]

access-list <acl_id> permit|deny ... [log [disable|default] | [<level>] [interval <secs>]]






An Introduction to Show Commands

Show cpu usageShow xlate <detail>Show conn <detail>

Show InterfaceShow traffic

CLI tool used to extract

information from the PIX for

informationor troubleshooting

Used to monitor thehealth

of the PIX and draw a baseline

for your network

Displays current and past info related to the

PIXShow PerfmonShow Blocks

Show Memory Show processes


“Show Conn” and “Show Conn <detail>”

pixfirewall (config) # show connection2 in use, 2 most usedTCP out 192.150.49.10:23 in 10.1.1.15:1026 idle 0:00:22

Bytes 1774 flags UIOUDP out 192.150.49.10:31649 in 10.1.1.15:1028 idle 0:00:14

Flags D-

pixfirewall (config) # show connection detail2 in use, 2 most usedFlags: A – awaiting inside ACK to SYN, a – awaiting outside ACK to SYN,

B – initial SYN from outside, D – DNS, d – dump,E – outside back conection, F – outside FIN, f – inside FIN,G – group, H – H.323, I – inbound data, M – SMTP data,m – SIP media, O- outbound data, P – inside back connection,q – SQL*Net data, R – outside acknowledged FIN,R – UDP RPC, r – inside acknowledged FIN, S – awaiting inside SYN,s – awaiting outside SYN, T – SIP, t – SIP transient, U – up

TCP outside: 192.150.49.10/23 inside:10.1.1.15/1026 flags UIOUDP outside: 192.150.49.10/31649 inside:10.1.1.15/1028 flags dD



Connection Flags

Reset-I Reset was from inside

Reset-O Reset was from Outside

TCP FINs Normal close down sequence

FIN Timeout Force termination after 15 seconds

SYN Timeout Force termination after 2 min

Xlate Clear Command line removal

Deny Terminate by appln inspection

SYN Control Back Channel Initiation frm wrong side

Uauth Deny Deny by URL Filter

Unknown Catch all error

Connection Termination Reasons

U UP

f Inside FIN

F Outside FIN

r Inside ACK FIN

R Outside ACK FIN

s Awaiting Outside SYN

S Awaiting Inside SYN

M SMTP Data

I Inbound Data

O Outbound Data

Snip of Connection flags and its interpretation


“Show Xlate” and “Show Xlate <detail>”

pixfirewall (config) # show xlate3 in use, 3 most usedPAT Global 192.150.49.1(0) Local 10.1.1.15 ICMP id 340PAT Global 192.150.49.1 (1024) Local 10.1.1.15(1028)PAT Global 192.150.49.1 (1024) Local 10.1.1.15(516)

pixfirewall (config) # show xlate detail3 in use, 3 most usedFlags: D –DNS, d – dump, I – identity, I – inside, n – no random,

o – outside, r – portmap, s – staticTCP PAT from inside:10.1.1.15/1026 to outside:192.150.49.1/1024 flags riUDP PAT from inside:10.1.1.15/1028 to outside:192.150.49.1/1024 flags riICMP PAT from inside:10.1.1.15/21505 to outside:192.150.49.1/0 flags ri



Xlate Flags

s Static translation slot

d Dump translation slot on next clearing cycle

r Port map translation

n No randomization of TCP sequence number

o Outside address translation

i Inside address translation

D DNS A RR rewrite

I Identity translation from NAT 0

FLAG DESCRIPTION


Show CPU Usage

First introduced in PIX OS version 6.0(1)

Under normal conditions the PIX CPU should stay below 30% (baseline as perNetwork and Pix Model). If the CPU reaches 100% the PIX will start dropping packets

The show cpu usage command displays the CPU over time as a running average.

A Note

An Examplepixfirewall# show cpu usageCPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1%

The percentage usage prints as NA (not applicable) if the usage is unavailable for the specified time interval. This can happen if the user asks for CPU usage before the 5-second, 1-minute, or 5-minutes



Show Traffic

The show traffic command displays the traffic, in packets and in bytes, out each interface of the PIX.

An Example

pixfirewall# show traffic outside:

received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec

transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec

inside: received (in 124.650 secs):

261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec

transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec


Show Blocks

The show blocks command, along with the show cpu usage command, are useful in determining whether the PIX is being overloaded.

The blocks are internal storage locations, similar to queues on a router. A packet is stored in a block until the PIX can process it and place it on the outbound interface xmit queue.

pixfirewall# show blocks

SIZE MAX LOW CNT4 1600 1597 1600 80 400 399 400 256 500 495 499 1550 1444 1170 1188 16384 2048 1532 1538

An Example



Show Local-Host

The show local-host command displays the translation and connection slots for all local hosts.

The clear local-host command stops traffic on all local hosts. The clear local-host <ip_address> command stops traffic on the local host specified by its IP address

show local-host 10.1.1.15 local host: <10.1.1.15>, conn(s)/limit = 2/0, embryonic(s)/limit = 0/0 Xlate(s): PAT Global 172.16.3.200(1024) Local 10.1.1.15(55812) PAT Global 172.16.3.200(1025) Local 10.1.1.15(56836) PAT Global 172.16.3.200(1026) Local 10.1.1.15(57092) PAT Global 172.16.3.200(1027) Local 10.1.1.15(56324) PAT Global 172.16.3.200(1028) Local 10.1.1.15(7104) Conn(s): TCP out 192.150.49.10:23 in 10.1.1.15:1246 idle 0:00:20 Bytes 449 flags UIO TCP out 192.150.49.10:21 in 10.1.1.15:1247 idle 0:00:10 Bytes 359 flags UIO

An Example


Show Tech-Support Enhancements (6.3)

• The show tech output was enhanced to include some additional “show” commands that can be used to troubleshoot memory and performance issues.

• On the right are the commands included in the show tech output. Note: They are in order. (New commands are in red)

show versionshow clockshow memoryshow conn countshow xlate countshow blocksshow interfaceshow cpu usageshow processshow failovershow trafficshow perfmonshow running-config

show versionshow clockshow memoryshow conn countshow xlate countshow blocksshow interfaceshow cpu usageshow processshow failovershow trafficshow perfmonshow running-config



Show Output Filters

Output filters have been added to PIX 6.3, similar to the ones in IOS. To use them, at the end of show <command>, use the pipe character '|' followed by:

begin|include|exclude|grep [-v] <regular_exp>to filter the show output.

Begin - start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output.

include – Display any line that matches the RegExexclude – Display any line that does not match the RegExgrep – same as includegrep –v – same as exclude

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>show <cmd> | begin|include|exclude|grep [-v] <regular_exp>


Show Output Filters

Note: You must include a space on either side of the pipe for the command to be accepted. Also, trailing spaces are counted.

EX:• Display the interface stats starting with E2

show interface | begin ethernet2• Display only the route statements

show run | include route• Display the config, except for the access-lists

show run | exclude access-list• Display the access-list entries that contain address 10.1.1.5

show access-list | grep 10.1.1.5• Display only access-list entries that have hitcounts

show access-list | grep –v hitcnt=0

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>show <cmd> | begin|include|exclude|grep [-v] <regular_exp>






Output Interpreter

Great tool to catch common configuration errors.

Select the outputIn question

Paste the output



Output Example

Snip of Output

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

Example Of Messages






What Is PDM ?GUI interface to the PIX, designed to manage, monitor and troubleshoot the PIX

First available with PIX Version 6.0

The PDM image is loaded independently from the PIX OS image

Excellent tool for secure remote device management of the PIX


PDM FAQ’s

PDM offers a CLI option within the GUI

Can I access the CLI thru PDM?

Display’s Syslog within the GUI, and Show comands via the CLI optionCan I view Syslogs and Show commands ?

Will let you view information in a graphical form, thus letting you build a baseline for activity to and thru the PIX .

How is it different from CLI as far as troubleshooting is concerned

Graphs can be book marked with your settings or exported for further analysis

Can I save this graphs or export them?

User will have SSL access to the PIXIs it secure?



Show Xlate as it Would Appear Graphically


PDM as a Monitoring Tool



Agenda








Common Issues

• PIX Common IssuesAccessing the InternetAccessing Internal Network from the InternetIssues with traffic between interfaces PIX Not RedirectingFailover

• FWSM Common IssuesConfiguration IssuesPassing Traffic OutboundUnderstanding Failover



Accessing the Internet

Internet

BOB

Web Server

INSIDE OUTSIDE10.10.10.x

http://www.xyz.com

Problem:

Accessing The Internet

Troubleshooting:

Translation

Routing

Access-list applied incorrectly


PIX needs to be told a translation method to pass outbound traffic

PIX needs to be told a translation method to pass outbound traffic

global (outside) 1 interfacenat (inside) 1 10.10.10.0 255.255.255.0

global (outside) 1 192.168.1.1 – 192.168.1.253nat (inside) 1 10.10.10.0 255.255.255.0

Verify Translation Commands

static (inside,outside) 192.168.1.254 10.10.10.1 netmask 255.255.255.255

http://www.xyz.com

Internet

Web Server

INSIDE OUTSIDE10.10.10.x

.1

.2

.3

Translation



A Note on Translation

NAT 0 and Access-Lists

Static translations take precedence over a NAT/global

translation

Perform a clear xlate if Changes have been made

to global pools or static statements

Do not overlap IP addresses between

statics andglobal pools


Verified Translation11

Verify Route Commands on the PIX

• Check to make sure the PIX has the correct default gateway

pixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2

• If trying to access the internet from behind a layer 3 device, verify the PIX has a route to that network

pixfirewall (config)# route inside 172.16.171.0 255.255.255.0 10.10.10.2

172.16.171.x

.2

10.10.10.x

Internet

Web Server

INSIDE OUTSIDE

.1

63.1.1.x

.1 .2

http://www.xyz.com.3

Routing



11 Verified Translation

22 Verified Routing Information

.2

10.10.10.x

Internet

Web Server

INSIDE OUTSIDE

.1

63.1.1.x

.1 .2

http://www.xyz.com.3

Access-Lists

Verify if any access-lists are applied

pixfirewall (config) # access-list acl permit tcp host 10.10.10.3 any eq telnet

pixfirewall (config) # access-group acl in interface inside

If you have an access-list applied on the inside interface, check to make sure traffic is permitted outbound. Remember, there is an implicit deny at the endof an access-list

Note


Common Issues





Accessing Internal Network From Internet

Troubleshooting:

Translation RoutingAccess-list applied incorrectly

Web Server 10.10.10.x

InternetINSIDE OUTSIDE

.1

63.1.1.x

.1 .2.1

http://www.xyz.com

Problem:Internal Web Server Not Accessible to Users on the Internet


http://www.cisco.com

.2

10.10.10.x

InternetINSIDE OUTSIDE

.1

63.1.1.x

.1 .2.1

A Static Translation is required to pass inbound trafficA Static Translation is required to pass inbound traffic

pixfirewall(config ) #static(inside,outside) 63.1.1.3 10.10.10.1 netmask 255.255.255.255

Verify Translation Commands

Example of a Syslog Message With No Static Defined305005: No translation group found for tcp src outside:200.1.1.1/35550 dst inside:63.1.1.3/80

Translation



• Check to make sure the PIX has the correct default gatewaypixfirewall(config)# route outside 0.0.0.0 0.0.0.0 63.1.1.2

• Verify the PIX has a route to the internal networkpixfirewall(config)# route inside 172.16.171.0 255.255.255 10.10.10.2

• Other Issues to Consider• Confirm default gateway on your Web Server• Verify your layer 3 device is routing correctly

http://www.xyz.com

172.16.171.x.2

10.10.10.x

Internet

Web Server

.1

63.1.1.x

.1 .2

.2 .1

INSIDE OUTSIDE

Routing Issues


Traffic has to be explicitly allowed into the PIX from a lower security to a higher security

Traffic has to be explicitly allowed into the PIX from a lower security to a higher security

• Check to make sure you have permitted interesting traffic explicitlypixfirewall (config) # access-list acl permit tcp any host 63.1.1.5 eq http

pixfirewall (config) # access-group acl in interface outside

• If you have an access-list applied, check to make sure traffic is permitted inbound. Remember, there is an implicit deny at the end of an access-list

Access-Lists

172.16.171.x.2

10.10.10.x

Internet

Web Server

.1

63.1.1.x

.1 .2

.2 .1

INSIDE OUTSIDE

http://www.xyz.com



Common Issues

• PIX Common IssuesAccessing the InternetAccessing Internal Network from the InternetIssues with traffic between interfacesPIX Not RedirectingFailover



Issues with Traffic Between Interfaces

11

11 Static and Access-list/conduit

DMZ 1 (40)

INSIDE (100)

DMZ (30)

OUTSIDE (0)

Static and Access-list/conduit

22

22

Static or a NAT/Global Statement

33

33



Common Issues




PIX Is NOT Redirecting

172.16.171.x

.2

10.10.10.xInternet

INSIDE OUTSIDE

.1

63.1.1.x

.1 .2

http://www.xyz.com

.1

11 PIX will not handle redirects

22 Change user’s default gateway to be the layer 3 device

33 Modify the layer 3 device’s default gateway to be the PIX



Common Issues




PIX Failover

PIX Failover Cable

Active Unit Standby Unit

• If the primary PIX fails, the secondary will take over the duties of the primary to perform its functionality

• Identical hardware and software is required for failover

• A mechanism to provide high availability in your network



Failover On

Cable status: Normal

Reconnect timeout 0:00:00

This host: Primary - Standby (Failed)

Active time: 7140 (sec)

Interface 0 (192.168.1.1): Normal (Waiting)

Interface 1 (172.16.171.54): Failed (Waiting)

Other host: Secondary - Active

Active time: 30 (sec)



pixfirewall (config)# show failover

Commands to Verify Failover Is Active


Network InterfStatus Test

Network ActivityTest

ARP Test

Ping Test

Power Failure

Failover Cable Failure

Why Will Failover Happen?



How to Verify Why it Failed Over

No proper ACK for 15+ seconds after a command has been sent on the serial cable.

3

An interface did not pass one of the 4 failover tests. 2

No Failover hello seen on Serial cable for 30 + seconds. This ensures that Failover is running properly on the other PIX.

1

If you see the following syslog message after a failover:

%PIX-1-103001: (Secondary) No response from other firewall (reason code = code).

and you want to know what the reason code is, look at the table below.


Best Practices for Failover

• Hardcode transmission speed and duplex settings on PIX and switch ports

PIX:

‘interface ethernet0 100full’

Cat6K:

‘set port speed 3/25 100’

‘set port duplex 3/25 full’

• Enable Port Fast

• Turn off Trunking and Channeling

• Licensing Issues



Common Issues

• PIX Common IssuesAccessing the InternetAccessing Internal Network from the InternetIssues with traffic between interfaces PIX Not RedirectingFailoverPassword Recovery



Password Recovery

• For Password Recovery you need the PIX Password Lockout Utility

e.g. np63.bin for PIX 6.3 release

Each IOS release has its own password recovery file.

• You also need a TFTP server.

• You need to bring down the PIX into ROM Monitor mode.



Password Recovery

• Issue the following commands:

monitor> interface 0

monitor> address 10.21.1.99

monitor> server 172.18.125.3

monitor> file np63.bin

monitor> gateway 10.21.1.1

monitor> tftp

Do you wish to erase the passwords? [yn] y

Passwords have been erased.

Rebooting....

http://www.cisco.com/warp/public/110/34.shtmlOnline Resource


Common Issues


• FWSM Common IssuesConfiguration IssuesConnectivity IssuesUnderstanding Failover



Understanding the Configuration

FWSM

inside

Vlan 6010.60.1.0/24

Vlan 3010.30.1.0/24

10.30.1.1

10.60.1.1

outside

CatOS 7.5(1)

6K> (enable) set vlan 30 8/1,60 8/26K> (enable) set vlan 30,60 firewall 5vlans 30,60 declared secure for

firewall module 5

-----------------6K> (enable) session 5FWSM#conf tnameif 30 outside 0nameif 60 inside 100ip address outside 10.30.1.2/0ip address inside 10.60.1.1/24

Native IOS - 12.1(13)E

Router#config t!vlan 30,60firewall vlan-group 1 30,60firewall module 5 vlan-group 1!int fa8/1switchport access vlan 30

Int fa8/2switchport access vlan 60

-----------------FWSM#conf tnameif 30 outside 0nameif 60 inside 100ip address outside 10.30.1.1/0ip address inside 10.60.1.1/24

Slot 5

Fa8/1

Fa8/2


Vlan 30

Vlan 40Vlan 50

Vlan 60

FWSM

Firewall vlan-group 1 30,40,50,60Firewall module 5 vlan-group 1!Interface VLAN 30

ip address 10.30.1.1/24!Interface VLAN 70

ip address 10.70.1.1/24!session slot 5 processor 1nameif 30 outside 0nameif 40 dmz1 40nameif 50 dmz2 50nameif 60 inside 100ip address outside 10.30.1.2/0route outside 0/0 10.30.1.1ip address inside 10.60.1.1/24jp address dmz1 10.40.1.1/24ip address dmz2 10.50.1.1/24

outside

Vlan 70

dmz1dmz2

inside

10.70.1.0/24

10.30.1.0/24

10.40.1.0/2410.50.1.0/24

10.60.1.0/24

Catalyst 6500

FWSM and the MSFC

MSFC

.1

.2

Internet



FWSM Key Configuration Issues

• RestrictionsA VLAN can belong to only one firewall groupA max of 100 VLANS supported per FWSMHidden VLANs or default VLAN cannot be configured as a firewall VLANOne SVI Rule

• GuidelinesRestrict Trunks to carry Firewall VLANDo not map the same Firewall group to two different blades (unless it is a redundant setup)VTP should be in transparent mode


Common Issues


• FWSM Common IssuesConfiguration IssuesConnectivity IssueUnderstanding Failover



FWSM – Did We Not Cover This?

959595SEC-3205203_05_2002_c1 95

Resolutionnameif vlan208 outside security0nameif vlan336 inside security100access-list letmein permit ip any any ip address outside 10.66.79.140 255.255.255.224ip address inside 10.1.1.1 255.255.255.0global (outside) 1 interfacenat (inside) 1 10.1.1.0 255.255.255.0 0 0access-group letmein in interface insideroute outside 0.0.0.0 0.0.0.0 10.66.79.129 1

Problem:Cannot Pass Traffic Outbound


Common Issues


• FWSM Common IssuesConfiguration IssuesConnectivity IssueUnderstanding Failover



Failover (Intra and Inter Chassis)

A dedicated logical interface (VLAN interface) is created for failover communications – uses failover protocol to detect a failure

Cat6K

FWSM FWSM

Cat6K Cat6K

FWSM

FWSM


FWSM Failover: Things to Note

• Configuration is performed on the primary unitThe primary unit is designated using the failover lan unitprimary command

• Assign the same IP address to that interface as the one you assigned to the primary unit

• Using the failover ip address command, assign the same address you used on the primary unit

• Unit will only fail over if over 50 % “interfaces” have failed (CSCea87456)



Agenda








IOS Firewall Platform Compatibility

• IOS Firewall was introduced in 12.0(5)T

• Flash and RAM requirements vary depending on the router platform

• Router platforms include:

Small Office: 800* and uBR900 series

Branch Offices: 1600, 2500, 2600, and 3600 series

WAN and high throughput: 7100, 7200, 7500, and RSM



What Are the Features of IOS Firewall?

Policy based multi-interface

supportJava blocking

Per-user authentication

and authorization

Basic and advanced traffic

filtering

Dynamic port mapping

Intrusion detection

Real time alerts and audit trail

DoS detection and prevention

Stateful packet inspection


CBAC Packet Flow

IPSecPkt?

No State IDSInput Int

AuthProxy

Inbound ACLInput Int

DecryptPacket

Save addrand port

if NAT cfg

StatelessIDS - Input

InboundInput ACL

NAT BeforeRouting

Routing NAT AfterRouting

No State IDSOutput Int

FragmentInspection

PktNATed?

FindPregen?

Outbound ACLOutput Int

CreateOutput ACL

CBAC andState based

IDS

IPSecPkt?

EncryptPacket

Y

N

YY

N N

Y

N



Agenda








Best Tool Is to Understand How CBAC Works

Why Access-ListsWhat Do I Inspect? and

or

Are these dynamic holes ACL’s safe?

Why do I Inspect? and

Internal Network External Networke0 e1

Can this crash my router Do I inspect all interfaces



What Is CBAC

Allow Return TrafficDeny Traffic Initiating from Outside

Inside Outside

access-list 101 deny ip any anyinterface ethernet1ip access-group 101 in

ip inspect name foo tcpinterface ethernet0ip inspect foo in

SYNSYN + ACKACK

access-list 101 permit tcp host B eq b host A eq aaccess-list 101 deny ip any anyinterface ethernet1ip access-group 101 in

access-list 101 permit tcp host B eq b host A eq aaccess-list 101 deny ip any anyinterface ethernet1ip access-group 101 in

A:aA:aA:a

B:bB:bB:b


Access-List Guidelines

• Configure access lists including entries that permit certain ICMP traffic from unprotected networks; routers will need to see certain ICMP traffic from these networks

• Access lists must explicitly permit ICMP messages from the outside to the inside as CBAC will not currently inspect ICMP traffic for return packets

• Anti-spoofing protection with an access list denying network traffic from a source address that matches an address on the protected network

• Add an entry to block broadcasts

Things to know:



Show Commands

show ip access-listshow ip inspect name inspection-nameshow ip inspect configshow ip inspect configshow ip inspect interfaces show ip inspect session [detail] show ip inspect allshow ip inspect stathttp://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/iosfw2/iosfw2_2.htm#12583


Debugging CBAC

Generic Debugdebug ip inspect object-creationdebug ip inspect object-deletiondebug ip inspect eventsdebug ip inspect timersdebug ip inspect detail

Audit Trails

ip inspect audit-trail

Transport Level Debugsdebug ip inspect tcpdebug ip inspect udp

Application Protocol Debugs

debug ip inspect protocol



Common Debugging Techniques

• Study the logic of your list or try defining an additional broader list:

access-list # permit tcp any any access-list # permit udp any any access-list # permit icmp any any int <interface> ip access-group # in|out

•Use an extended access-list with a log option at the end:

access-list 101 deny ip host 171.68.118.100 host 10.31.1.161 log access-list 101 permit ip any any

Mar 1 04:44:19.446: %SEC-6-IPACCESSLOGDP: list 111 permitted icmp 171.68.118.100 -> 10.31.1.161 (0/0), 15 packets*Mar 1 03:27:13.295: %SEC-6-IPACCESSLOGP: list 118 denied tcp 171.68.118.100(0) -> 10.31.1.161(0), 1 packet

If access-list may be a suspect:

• If the ip inspect list is suspect, try debug ip inspect <type_of_traffic> :

Feb 14 12:41:17 10.31.1.52 56: 3d05h: CBAC* sis 258488 pak 16D0DC TCP P ack3195751223 seq 3659219376(2) (10.31.1.5:11109) => (12.34.56.79:23)


Agenda









Configuration Issues

The Most Common Configuration Error Is the Direction of Inspection

The Most Common Configuration Error Is the Direction of Inspection

Access List Inbound on e1Inspect Inbound on e0 and

or

Access List Inbound on e1Inspect Outbound on e1 and

Protected Network Unprotected Networke0 e1


IOS FW Dropping Packets

• Base Line Your Network

• Adjust Your Threshold Values As Needed

• Check Your Access-Lists

• Verify Your Inspect Statements

• Check for Asymmetrical Routing



Tips for Troubleshooting CBAC

If traffic is being denied:

See if an access-list is not denying traffic. Remove the access-group and see if traffic in question is permitted. If possible apply extendedaccess-lists.

Debugs on the Router:

Log your deny statements temporarily

Router(config)#ip access-list extended IOSFWRouter(config-ext-nacl)#deny ip any any log

CBAC related debugs will give a lot of information if CBAC is working the way it is supposed to be and return traffic is permitted

Debug IP Packet Detail:Router(config) # access-list 101 tcp host 10.1.1.1 host 192.168.1.1Router # debug ip packet detail 101


Summary

• Understand How PIX Passes Traffic

• Security Levels

• Troubleshoot With The Right Tools

• Difference between the FWSM and PIX

• How does CBAC Work

• Online Resources

• TAC !



Some Good Links

PIX Firewallhttp://www.cisco.com/univercd/cc/td/doc/product/iaabu/PIX/PIX_sw/v_63/index.htmhttp://www.cisco.com/cgi -bin/tablebuild.pl/PIXhttp://www.cisco.com/pcgi -bin/Support/browse/psp_view.pl?p=Hardware:PIX&s=Software_Configurationhttp://www.cisco.com/pcgi -bin/Support/browse/psp_view.pl?p=Hardware:PIX&s=Troubleshooting#Known_Problems

FWSMhttp://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/78_14450.htm

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/fwsm/index.htm

http://www.cisco.com/en/US/products/hw/modules/ps2706/products_qanda_item09186a00800c4fee.shtml

IOS FWhttp://www.cisco.com/pcgi -bin/Support/browse/psp_view.pl?p=Software:Cisco_IOS_Firewall&s=Implementation_and_Configuration

http://www.cisco.com/warp/partner/synchronicd/cc/pd/iosw/ioft/io fwft/prodlit/fire_qa.htm

http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/iosfw2/iosfw2_2.htm#xtocid135950


Please Complete Your Evaluation Form

Session SEC-3020

cisco firewall

Documents