2010 cpug con tobias lachmann check point troubleshooting

5/24/2018 2010 CPUG CON Tobias Lachmann Check Point Troubleshooting

1/79

Check Point TroubleshootingOops! Its not working!


2/79

Introduction

Troubleshooting is more or less the same sinceyears

The great How to use fw monitor document is from2003 still valid!

Some minor changes to buffer size, command lineoptions

New kernel modules introduced with R70 and R71,but no information officially available

We have to stick with the old stuff


3/79

How to approach troubleshooting

Collect information

What is the problem? What are the symptoms?

Can the problem be replicated?

Random occurence?

Anything changed in the setup?

User-related or machine-related?

List systems that are part of the conversation


4/79

Bug or configuration problem?

Common configuration problems:

Firewall rule prevents traffic

SmartDefense / IPS blade prevents traffic

Antispoofing

misconfigured routing

wrong encryption domain

wrong username / password



5/79


Any reference for problem or error message?

official documentation

SecureKnowledge

CPUG forum

Check Point forum

Google


6/79

fw ctl zdebug drop

Replicate the problem and have a look at the gateway:

fw ctl zdebug drop

lists all dropped packets in realtime

gives an explanation why the packet is dropped

fw_log_drop: Packet proto=6 81.63.88.122:2720 ->

212.1.52.64:445 dropped by

fw_handle_first_packet Reason: Rulebase drop -rule 12;

Why is it called zdebug? Developed by Tamir Zegman.


7/79

Firewall status

Current connections?

fw tab t connections s

[Expert@firewallr70]# fw tab -t connections -s

HOST NAME ID #VALS #PEAK #SLINKS

localhost connections 8158 1 1 1

fw ctl pstat | grep Connections

[Expert@firewallr70]# fw ctl pstat | grep Connections

Concurrent Connections: 0% (1 out of 24900) - below

low watermark


8/79

ClusterXL

Status information

fw hastat

HOST NUMBER HIGH AVAILABILITY STATE MACHINE STATUSlocalhost 2 stand-by OK

cphaprob state

Cluster Mode: New High Availability (Primary Up)

Number Unique Address Assigned Load State

1 192.168.55.202 100% Active

2 (local) 192.168.55.201 0% Standby


9/79

ClusterXL

Displays ClusterXL Devices

cphaprob ia list

Displays physical and cluster interfaces

cphaprob a if

Statistics of ClusterXL sync

fw ctl pstat

cphaprob syncstat

Reset statistics of ClusterXL sync

cphaprob reset syncstat


10/79

Licenses

Limited number of hosts?

fw lichosts

Count of used hosts

fw lichosts | wc l

SecureClient licenses used

dtps lic


11/79

Licenses

Show license

cplic print

Compare to SmartUpdate / SmartView Monitor output

Especially UTM products sometimes tend to mess up

with licenses which can cause Antivirus, Antispamor URL filtering to stop working

You need to keep contracts updated!

Use evaluation licenses for testing!


12/79

Content scanning

Verify update process of Antivirus or URL filteringusing avsu_client command

avsu_client app URL Filtering

fetch_remote fi

for fetching the index file (signatures up-to-date?)

avsu_client app URL Filtering

fetch_remote fe

for fetching entitlement / signatures


13/79

fw monitor

What is it?

fw monitor command triggers a Check Point

kernel module that is used to capture packets.

What makes it different?

Packet capture at multiple positions within the kernelmodule chain, both for inbound and outboundpackets. It doesnt work on Layer-2, so no MACaddresses are shown in the output.

fw monitor is available on all platforms.


14/79

fw monitor

What makes it different?

filters packets using INSPECT code

sees packets with the eyes of the gateway

Shows flow of packets through the gateway

No Layer-2 information in capture files


15/79

fw monitor

IP Routing IP

VM

NIC

VM

NIC

TCP

App.

TCP

App.

post-inbound (I)

pre-inbound (i) post-outbound (O)

pre-outbound (o)


16/79

fw monitor

[ Exper t @f w1] # f w moni t or - e "accept ( sr c=212. 1. 52. 68 ordst =212. 1. 52. 68) ; "

moni t or : get t i ng f i l t er ( f r om command l i ne)moni t or : compi l i ng

moni t or f i l t er :

Compi l ed OK.moni t or : l oadi ngmoni t or : moni t or i ng ( cont r ol - C t o st op)

et h3. 7: i [ 52] : 212. 1. 56. 233 - > 212. 1. 52. 68 ( TCP) l en=52 i d=18406TCP: 56661 - > 22 . S. . . . seq=b2f 3509d ack=00000000et h3. 7: I [ 52] : 212. 1. 56. 233 - > 212. 1. 52. 68 ( TCP) l en=52 i d=18406TCP: 56661 - > 22 . S. . . . seq=b2f 3509d ack=00000000

et h0: o[ 52] : 212. 1. 56. 233 - > 212. 1. 52. 68 ( TCP) l en=52 i d=18406TCP: 56661 - > 22 . S. . . . seq=b2f 3509d ack=00000000et h0: O[ 52] : 212. 1. 56. 233 - > 212. 1. 52. 68 ( TCP) l en=52 i d=18406TCP: 56661 - > 22 . S. . . . seq=b2f 3509d ack=00000000et h0: i [ 52] : 212. 1. 52. 68 - > 212. 1. 56. 233 ( TCP) l en=52 i d=0TCP: 22 - > 56661 . S. . A. seq=68a919c9 ack=b2f 3509eet h0: I [ 52] : 212. 1. 52. 68 - > 212. 1. 56. 233 ( TCP) l en=52 i d=0

TCP: 22 - > 56661 . S. . A. seq=68a919c9 ack=b2f 3509eet h3. 7: o[ 52] : 212. 1. 52. 68 - > 212. 1. 56. 233 ( TCP) l en=52 i d=0TCP: 22 - > 56661 . S. . A. seq=68a919c9 ack=b2f 3509eet h3. 7: O[ 52] : 212. 1. 52. 68 - > 212. 1. 56. 233 ( TCP) l en=52 i d=0TCP: 22 - > 56661 . S. . A. seq=68a919c9 ack=b2f 3509e


17/79

fw monitor

et h3. 7: O[ 52] : 212. 1. 52. 68 - > 212. 1. 56. 233( TCP) l en=52 i d=0

TCP: 22 - > 56661 . S. . A. seq=68a919c9ack=b2f 3509e


18/79

fw monitor

fw monitor options overview

-u | s Shows UUID or SUUID for every packet-i write data to STDOUT

-d | D debug / more debug output-e filter for expression (CLI mode)-f read filter expression from file-l limit length of captured packet-m which positions should be shown-x print raw packet data-o write packet into file-p|x| insert fw monitor at specific chain position-p all insert fwmonitor between all kernel modules-ci stop capture after count incoming packets-co stop capture after count outgoing packets


19/79

fw monitor


20/79

fw monitor

fw monitor -e "accept [9:1]=1;

Capture only ICMP packets


21/79

fw monitor

Capture only packets from a special host

fw monitor -e "accept [12,b]=192.168.1.1;


22/79

fw monitor

Filtering will be easier for you if you use macros.

Macros for fw monitor are defined in$FWDIR/lib/fwmonitor.def which references$FWDIR/lib/tcpip.def, where the actual

expression is located.

Example: filter for source IP

fwmonitor.def macro = src

tcpip.def macro = ip_srcexpression = [12,b]


23/79

fw monitor

Use macros together with operators to addcomplexity:

accept (src=x.x.x.x or dst=x.x.x.x)

accept ((src=x.x.x.x, dst=y.y.y.y) or(src=y.y.y.y, dst=x.x.x.x));

accept not (sport=22 or dport=22);

accept sport=21 and not (src=x.x.x.x);


24/79

fw monitor

Use fw monitor to see if packets are translated

fw monitor -e accept (src=212.1.56.151 or dst=212.1.56.151);

eth0:i[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053eth0:I[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053

eth1:o[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053

eth1:O[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053

fw monitor -e accept (src=212.1.56.151 or dst=212.1.56.151);

eth0:i[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=31171

eth0:I[48]: 212.1.56.151 -> 192.168.199.2 (TCP) len=48 id=31171

eth1:o[48]: 212.1.56.151 -> 192.168.199.2 (TCP) len=48 id=31171

eth1:O[48]: 212.1.56.151 -> 192.168.199.2 (TCP) len=48 id=31171


25/79

fw monitor

Common expressions for fw monitor

fw monitor e accept (src=x.x.x.x or

dst=x.x.x.x);

fw monitor m iO e accept host(x.x.x.x);

fw monitor e accept ((src=x.x.x.x, dst=y.y.y.y)or (src=y.y.y.y, dst=x.x.x.x));

fw monitor e accept (ip_p=x);

Combine with o for output into a file.

Inspect Code Generator: http://decock.org/ginspect/


26/79

fw monitor

Read complex expressions from a filter file:

fw monitor f

If you use macros in a filter file, make sure toinclude the appropriate definition file.

#include fwmonitor.def

accept ((sport=22 or dport=22) and not

(host(x.x.x.x));


27/79

Use for better analysis of capture files.

Preferences Protocols Ethernet Check boxAttempt to interpret as Firewall-1 monitor filePreferences Protocols FW-1 Activate UUID,chain position, summary in protocol tree

Add column fw1 chain of format FW-1 monitor if/direction

Add coloring rules

preIn Filter String fw1.direction == ipostIn Filter String fw1.direction == IpreOut Filter String fw1.direction == opostOut Filter String fw1.direction == O

fw monitor


28/79

fw monitor

On UTM-1 Edge

Setup Tools Packet Sniffer

two modes: normal sniffer or fw monitor

On SecuRemote/SecureClient

srfw monitor o


29/79

Troubleshooting UTM-1 Edge

Analyse local policy

Run info fw rules on command line

or WebUI Setup Tools Command Line

Analyse NAT policy

Run info nat on command line

orWebUI Setup Tools Command line


30/79


Create diagnostics file

Log into WebUI

Setup Tools Diagnostics


31/79


Is the SMS process running on SmartCenter?

ps aux | grep sms

Is traffic reaching the SmartCenter?

fw monitor

libsw must be current, at least same version aslatest firmware installed on a Edge.

Check /opt/CPEdgecmp-R71/libsw/version.txt

[Expert@fwm]# head -n1 version.txtlibsw built with version 8.1.21


32/79


Sofaware Management Server Console

http://:9283/

- restart SMS

- reload SMS settings

- force policy update- reboot

- reset local (Edge) password

- view status information


33/79



34/79


Debugging Sofaware Management Server

Edit $FWDIR/conf/sofaware/SWManagement.ini

Change in line containing LogPolicy1 the valueInfo to Debug

Smsstop

sms confdir $FWDIR/conf/sofaware

Replicate the problem and watch for console output.

Terminate programm and restart SMS afterwards

smsstart


35/79


Configuration for Edge Devices on SPLAT under/opt/CPEdgecmp-R71/tmp

.pf ruleset.pfz compressed ruleset.topo topology for VPN.tpz compressed topology.p12 PKCS#12 certificate

Delete files. Install policy again to re-generate them.Make sure, that the files are compiled and the Edge

gets the latest version.


36/79

Opening a service request

Submit info to Check Point TAC or your CCSP/CSP

provide contact info

describe Check Point environment

list used gateway hardware

provide info about network topology and hardware

describe the problem / the symptoms in detailwhat kind of business impact has this problem

recommendation: get your supporter on the phoneand be available for remote sessions

use chat tool!


37/79

Opening a service request

Create compressed CPInfo diagnostic file

/opt/CPinfo-10/bin/cpinfo z

Create compressed CPInfo diagnostic file including logs

/opt/CPinfo-10/bin/cpinfo l z

CPInfo files can be viewed using InfoView

Make sure to have the latest CPinfo build installed!

Check sk30567 for instructions!


38/79

TAC organisation

Director TAC

INTL Support Escalations Diamond Services

3 Product TeamsHigh end

Core

VPN

Knowledge Center

3 Product TeamsHigh end

Core

VPN

Data Security

Escalation

Secure Knowledge Technical

Publications

Customer Focus

Programmers


39/79

TAC escalation

Support desk

Product team

Escalations

Customer focus programmer


40/79

TAC escalation path

http://www.checkpoint.com/services/contact/escalation.html


41/79

General debugging

kernel mode user mode

rtmmod

simmod

vpntmodvpnmod

fwmod

usbcore

security server

smscpd

fwd

fwm

fw, VPN, FG-1, H323,BOA, WS, CPAS, CLUSTER,RTM, kiss, kissflow, multik,

SFT, CI


42/79

kernel mode debug

View kernel modules with fw ctl debug h

kiss ??????

kissflow ???????fw "Firewall Module"

h323 "VoIP H.323 Module"

multik "related to CoreXL"

BOA "Malicious Code Protection Module"WS "SmartDefense Web Intelligence Module"

CI Content Inspection

CPAS "Active Streaming Module"

VPN "VPN Module"

RTM "SmartView Monitor Module"

SFT ???????

Cluster "ClusterXL Module"

FG-1 "Floodgate-1 QoS Module"

k l d d b


43/79

kernel mode debug

Some examples for modules and options:

Module: fw

Options: error warning cookie crypt domain exdriver filter hold if install ioctl kbufld log machine memory misc packet q xlatexltrc conn synatk media align balancechain bridge tcpstr scv ndis packval syncipopt link nat cifs drop

Module: vpn

Options: driver err packet policy sas rdpclear cipher init sr comp xl counters mspicphwd ref vin cluster nat l2tp warn

k l d d b


44/79

kernel mode debug

fw ctl debug

Allocation of a buffer for the debug logs

fw ctl debug buf [size in kb]

The main debug command

fw ctl debug m

Writing the debug logs into a file

fw ctl kdebug T f o

Stop debugging

fw ctl debug 0

k l d d b


45/79

kernel mode debug

Filter debug, only lines with in it arewritten to the output (best practice: error, failed)

fw ctl debug d

Filter debug, only lines that dont contain in it are written to the output

fw ctl debug d ^

Can be combined

fw ctl debug d error,failed,^packet

k l d d b


46/79

kernel mode debug

Stop debug messages when a certain string isissued.

fw ctl debug s

Example:

fw ctl debug s error

k l d d b


47/79

kernel mode debug

Example: debugging ClusterXL

fw ctl debug buf 32000

fw ctl debug m fw + conn drop packet ifsync

fw ctl debug m cluster all

fw ctl kdebug T f o

Example: debugging Site to Site VPN

fw ctl debug -buf 32000

fw ctl debug -m VPN allfw ctl debug -m fw + conn drop ld xlate

xltrc nat

fw ctl kdebug T f o

kernel mode debug


48/79

kernel mode debug

Example: debugging SIP


fw ctl debug m fw + conn drop vm sip

fw ctl kdebug T f o

Example: debugging VoIP


fw ctl debug -m fw + conn drop vm

fw ctl debug m h323 all

fw ctl kdebug T f o

kernel mode debug


49/79

kernel mode debug

Example: debugging SmartDefense


fw ctl debug m fw + conn drop vm tcp-strspii

fw ctl kdebug T f o

Example: debugging NAT


fw ctl debug -m fw + xlate xltrcfw ctl kdebug T f o

kernel mode debug


50/79

kernel mode debug

Example: debugging QoS


fw ctl debug m FG-1 all

fw ctl kdebug T f o

Example: debugging SmartView Monitor


fw ctl debug -m RTM all

fw ctl kdebug T f o

VPN debug


51/79

VPN debug

Best practice before starting debug

Compare configuration on both ends

often Phase I / Phase II parameters are not equalwhich causes the VPN to fail

take special notice of networks and subnet masks

carefully compare Pre-Shared-Secrets

Have a close look at the logs in SmartView Tracker

Most informations can be found in the logs

VPN debug


52/79

VPN debug

To determine status of VPN tunnels, use menu based

vpn tunnelutil vpn tu

or SmartView Monitor

To shutdown all VPN operation, use

vpn drv off

To enable VPN again, use

vpn drv on

install policy

VPN debug


53/79

VPN debug

VPN debugging events can be logged on thegateway

vpn debug on

Debug output is written to $FWDIR/log/vpnd.elg

More details can be logged using the command

vpn debug on TDERROR_ALL_ALL=5

Turn off debugging with

vpn debug off

VPN debug


54/79

VPN debug

IKE negotiations during VPN tunnel establishmentcan be logged in ike.elg

On the gateway:vpn debug ikeon / vpn debug ikeoff

Debug output is written to $FWDIR/log/ike.elg

VPN debug


55/79

VPN debug

Initiate VPN and IKE debug together

vpn debug trunc

Disable VPN and IKE debug

vpn debug off

vpn debug ikeoff

VPN debug


56/79

VPN debug

Capture traffic using fw monitor

fw monitor e accept port(500) or

port(4500); o monitor.out

Output file is monitor.out, IKE payloads areencrypted.

Capture traffic using vpn debug

vpn debug mon

Output file is ikemonitor.snoop, IKE payloads are in

clear.

Turn off with vpn debug moff.

VPN debug


57/79

VPN debug

On UTM-1 Edge appliance:WebUI -> Reports -> Tunnels -> save IKE trace

Click Save IKE Trace, which creates ike.elg

user mode debug


58/79

user mode debug

General syntax

fw debug TDERROR_ALL_ALL=fw debug OPSEC_DEBUG_LEVEL=

Exception: cpd

fwm debug


59/79

fwm debug

FWM controls connections from the SmartConsole tothe SmartCenter server and is responsible forpolicy related functions

To debug fwmdo the following

fw debug fwm on TDERROR_ALL_ALL=5fw debug fwm on OPSEC_DEBUG_LEVEL=9

To stop debug run

fw debug fwm off TDERROR_ALL_ALL=0fw debug fwm off OPSEC_DEBUG_LEVEL=0

Logs are written to $FWDIR/log/fwm.elg

fwm debug


60/79

g

[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]

fwnetobj_getbysicname:

table_chosen_get_with_param(eTABLE_NETWORK_OBJECTS,is_obj_SIC_name,

IP=212.1.56.233,CN=Gui_Client) returned NULL.Login failed: 212.1.56.233 is not allowed for remotelogin

[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]fwm_log: Login failed fromIP=212.1.56.233,CN=Gui_Client: Unauthorized client

Wed Sep 8 18:46:32 2010 (GMT): reject clientIP=212.1.56.233,CN=Gui_Client

[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]PM_policy_query: rule not found.

[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]PM_policy_query:

finished successfully. 1st method = deny

IP not defined in $FWDIR/conf/gui-clients

fwm debug


61/79

g


fwm_cpmi_auth_handler: authenticating admin admin by

Name and Password


Administrator admin found in fwm database[FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07]

CBinObjCommon::PackLogData: Field number:12, Data

offset:34, Type:eFtCstring,Value:Administrator

failed to log in: Wrong Password

fwd debug


62/79

g

FWD daemon controls logging, alerts,communication with the kernel, OPSECcommunication, invokes child processes (securityservers, ICA)

To debug fwddo the following

fw debug fwd on TDERROR_ALL_ALL=5

To stop debug run

fw debug fwd off TDERROR_ALL_ALL=0

Logs are written to $FWDIR/log/fwd.elg

Desktop log server debug


63/79

p g g

To debug dtls do the following

fw debug dtls on

To stop debug run

fw debug dtls off

Logs are written to $FWDIR/log/dtlsd.elg

Security servers debug


64/79

y g

Some examples for security servers:

FTP security server in.aftpd

Telnet security server in.atelnetdHTTP security server in.ahttpd

SMTP security server in.asmtpd

ClientAuth (900) in.ahclientdClientAuth (259) in.aclientd

AntiSpam security server in.msd

URL filtering security server in.aufpd

Security servers debug


65/79

Verify that security server process exists. Check$FWDIR/tmp for existing PID files.

Start debugging (example for FTP security server)

fw debug in.aftpd on FWAFTPD_LEVEL=3

Stop debugging

fw debug in.aftpd off FWAFTPD_LEVEL=3

cpd debug


66/79

CPD controls SIC, Policy install

To debug cpddo the following

cpd_admin debug on TDERROR_ALL_ALL=5

To stop debug run

cpd_admin debug off TDERROR_ALL_ALL=0

Logs are written to $CPDIR/log/cpd.elg

Secure Platform debug


67/79

Sometimes it is useful to verify file integrity andversion against a test environment, for exampleafter installation of ad-hoc fixes or HFA.

Usemd5sumfor creating hashes.[Expert@fwm]# md5sum upgrade_importe6c6417cca9db098b94673dd420a4903 upgrade_import

Use cpvinfo for displaying version information.

[Expert@fwm]# cpvinfo upgrade_import

Build Number = 730080036

Major Release = NGX

Minor Release = fli_up_ga

Release Number = 5.0.5

Version Name = NGX



68/79

For some problems with processes a core dump canbe usefull.

A core dump is a disk file that contains an image ofthe processs memory at the time of termination.

Core dumps are mainly used by Check Point R&D for

fixing a specific problem.

Handling Core Fileshttp://downloads.checkpoint.com/dc/download.htm?ID=10479



69/79

To enable core dumps do the following

ulimit c unlimited

um_core enable

Reboot

Check that /etc/sysconfig/enable_cores exist

after Reboot.

Dumps will be in /var/log/dump/usermode

Debugging GUI clients


70/79

Debug GUI clients

Dashboard fwpolicy.exe d o fwp_debug.txt

Tracker cplgv.exe d o cplgv_debug.txt

Monitor smartcons.exe d o smartcons_debug.txt

general syntax: -d o

Output is in specified directory or in

C:\Programme\CheckPoint\SmartConsole\R70\PROGRAM\data

if directory is omitted.

Resources


71/79

SmartSPLAT from ada Ulucan

www.smartsplat.com

Resources


72/79

Resources


73/79

Resources


74/79

Resources


75/79

Resources


76/79

Resources


77/79

fw monitor

http://www.checkpoint.com/techsupport/downloads/html/ethereal/fw_monitor_rev1_01.pdf

The CPinfo utility

https://supportcenter.checkpoint.com/supportcenter/porta

l?eventSubmit_doGoviewsolutiondetails=&solutionid=sk30567

Documents related to troubleshooting

http://blog.lachmann.org/2010/09/documents-related-to-troubleshooting/

Questions?


78/79

Still got a question?


79/79

Tobias Lachmann

[email protected]

http://blog.lachmann.org

2010 cpug con tobias lachmann check point troubleshooting

Documents

fw ctl zdebug dropreplicate

fw ctl zdebug droplists

fw monitor document

fw tab t connections

fw tab t connections

slinkslocalhost connections

firewall rule

packet reason