Check Point Troubleshooting Oops! Its not working!

Introduction
Troubleshooting is more or less the same since years The great How to use fw monitor document is from 2003 still valid! Some minor changes to buffer size, command line options New kernel modules introduced with R70 and R71, but no information officially available

We have to stick with the old stuff

How to approach troubleshooting

Collect information
What is the problem? What are the symptoms? Can the problem be replicated? Random occurence? Anything changed in the setup? User-related or machine-related? List systems that are part of the conversation

How to approach troubleshooting

Bug or configuration problem?
Common configuration problems:
Firewall rule prevents traffic SmartDefense / IPS blade prevents traffic Antispoofing misconfigured routing wrong encryption domain wrong username / password

How to approach troubleshooting

Any reference for problem or error message?
official documentation SecureKnowledge CPUG forum Check Point forum Google

fw ctl zdebug drop

Replicate the problem and have a look at the gateway:

fw ctl zdebug drop

lists all dropped packets in realtime gives an explanation why the packet is dropped
fw_log_drop: Packet proto=6 81.63.88.122:2720 -> 212.1.52.64:445 dropped by fw_handle_first_packet Reason: Rulebase drop rule 12;

Why is it called zdebug? Developed by Tamir Zegman.

Firewall status
Current connections?

fw tab t connections s
[Expert@firewallr70]# fw tab -t connections -s HOST NAME ID #VALS #PEAK #SLINKS localhost connections 8158 1 1 1

fw ctl pstat | grep Connections

[Expert@firewallr70]# fw ctl pstat | grep Connections Concurrent Connections: 0% (1 out of 24900) - below low watermark

ClusterXL
Status information fw hastat
HOST NUMBER localhost 2 HIGH AVAILABILITY STATE stand-by MACHINE STATUS OK

cphaprob state
Cluster Mode: Number 1 2 (local) New High Availability (Primary Up) Assigned Load 100% 0% State Active Standby

Unique Address 192.168.55.202 192.168.55.201

ClusterXL
Displays ClusterXL Devices
cphaprob ia list

Displays physical and cluster interfaces

cphaprob a if

Statistics of ClusterXL sync

fw ctl pstat cphaprob syncstat

Reset statistics of ClusterXL sync

cphaprob reset syncstat

Licenses
Limited number of hosts? fw lichosts Count of used hosts fw lichosts | wc l SecureClient licenses used dtps lic

Licenses
Show license

cplic print
Compare to SmartUpdate / SmartView Monitor output Especially UTM products sometimes tend to mess up with licenses which can cause Antivirus, Antispam or URL filtering to stop working You need to keep contracts updated! Use evaluation licenses for testing!

Content scanning
Verify update process of Antivirus or URL filtering using avsu_client command avsu_client app URL Filtering fetch_remote fi for fetching the index file (signatures up-to-date?) avsu_client app URL Filtering fetch_remote fe <[email protected]> <password> for fetching entitlement / signatures

fw monitor
What is it?
fw monitor command triggers a Check Point kernel module that is used to capture packets.

What makes it different?

Packet capture at multiple positions within the kernel module chain, both for inbound and outbound packets. It doesnt work on Layer-2, so no MAC addresses are shown in the output. fw monitor is available on all platforms.

fw monitor
What makes it different?
filters packets using INSPECT code sees packets with the eyes of the gateway Shows flow of packets through the gateway No Layer-2 information in capture files

fw monitor
App. TCP IP post-inbound (I) VM pre-inbound (i) NIC NIC VM post-outbound (O) Routing App. TCP IP pre-outbound (o)

fw monitor
[Expert@fw1]# fw monitor -e "accept (src=212.1.52.68 or dst=212.1.52.68);" monitor: getting filter (from command line) monitor: compiling monitorfilter: Compiled OK. monitor: loading monitor: monitoring (control-C to stop) eth3.7:i[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406 TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000 eth3.7:I[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406 TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000 eth0:o[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406 TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000 eth0:O[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406 TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000 eth0:i[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0 TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e eth0:I[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0 TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e eth3.7:o[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0 TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e eth3.7:O[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0 TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e

fw monitor
eth3.7:O[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0 TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e

fw monitor
fw monitor options overview
-u | s -i -d | D -e <expr> -f <file> -l <len> -m <mask> -x -o <file> -p|x| <pos> -p all -ci <count> -co <count> Shows UUID or SUUID for every packet write data to STDOUT debug / more debug output filter for expression (CLI mode) read filter expression from file limit length of captured packet which positions should be shown print raw packet data write packet into file insert fw monitor at specific chain position insert fwmonitor between all kernel modules stop capture after count incoming packets stop capture after count outgoing packets

fw monitor

fw monitor
Capture only ICMP packets

fw monitor -e "accept [9:1]=1;

fw monitor
Capture only packets from a special host

fw monitor -e "accept [12,b]=192.168.1.1;

fw monitor
Filtering will be easier for you if you use macros. Macros for fw monitor are defined in $FWDIR/lib/fwmonitor.def which references $FWDIR/lib/tcpip.def, where the actual expression is located. Example: filter for source IP
fwmonitor.def macro tcpip.def macro expression = = = src ip_src [12,b]

fw monitor
Use macros together with operators to add complexity: accept (src=x.x.x.x

or

dst=x.x.x.x)

accept ((src=x.x.x.x, dst=y.y.y.y) (src=y.y.y.y, dst=x.x.x.x));

or

accept

not

(sport=22

or

dport=22);

accept sport=21

and not

(src=x.x.x.x);

fw monitor
Use fw monitor to see if packets are translated
fw monitor -e accept (src=212.1.56.151 or dst=212.1.56.151);
eth0:i[48]: eth0:I[48]: eth1:o[48]: eth1:O[48]: 212.1.56.151 212.1.56.151 212.1.56.151 212.1.56.151 -> -> -> -> 195.244.116.166 195.244.116.166 195.244.116.166 195.244.116.166 (TCP) (TCP) (TCP) (TCP) len=48 len=48 len=48 len=48 id=27053 id=27053 id=27053 id=27053

fw monitor -e accept (src=212.1.56.151 or dst=212.1.56.151);

eth0:i[48]: eth0:I[48]: eth1:o[48]: eth1:O[48]: 212.1.56.151 212.1.56.151 212.1.56.151 212.1.56.151 -> -> -> -> 195.244.116.166 (TCP) len=48 id=31171 192.168.199.2 (TCP) len=48 id=31171 192.168.199.2 (TCP) len=48 id=31171 192.168.199.2 (TCP) len=48 id=31171

fw monitor
Common expressions for fw monitor
fw monitor e accept (src=x.x.x.x or dst=x.x.x.x); fw monitor m iO e accept host(x.x.x.x); fw monitor e accept ((src=x.x.x.x, dst=y.y.y.y) or (src=y.y.y.y, dst=x.x.x.x)); fw monitor e accept (ip_p=x);

Combine with o <file> for output into a file.

Inspect Code Generator: http://decock.org/ginspect/

fw monitor
Read complex expressions from a filter file: fw monitor f <filename> If you use macros in a filter file, make sure to include the appropriate definition file. #include fwmonitor.def accept ((sport=22 or dport=22) and not (host(x.x.x.x));

fw monitor
Use for better analysis of capture files.

Preferences Protocols Ethernet Check box Attempt to interpret as Firewall-1 monitor file Preferences Protocols FW-1 Activate UUID, chain position, summary in protocol tree Add column fw1 chain of format FW-1 monitor if/direction Add coloring rules
preIn postIn preOut postOut Filter Filter Filter Filter String String String String fw1.direction fw1.direction fw1.direction fw1.direction == == == == i I o O

fw monitor
On UTM-1 Edge Setup Tools Packet Sniffer

two modes: normal sniffer or fw monitor

On SecuRemote/SecureClient

srfw monitor o <filename>

Troubleshooting UTM-1 Edge

Analyse local policy Run info fw rules on command line or WebUI Setup Tools Command Line Analyse NAT policy Run info nat on command line or WebUI Setup Tools Command line

Troubleshooting UTM-1 Edge

Create diagnostics file Log into WebUI Setup Tools Diagnostics

Troubleshooting UTM-1 Edge

Is the SMS process running on SmartCenter? ps aux | grep sms Is traffic reaching the SmartCenter? fw monitor libsw must be current, at least same version as latest firmware installed on a Edge. Check /opt/CPEdgecmp-R71/libsw/version.txt [Expert@fwm]# head -n1 version.txt libsw built with version 8.1.21

Troubleshooting UTM-1 Edge

Sofaware Management Server Console http://<ip SmartCenter>:9283/
-

restart SMS reload SMS settings force policy update reboot reset local (Edge) password view status information

Troubleshooting UTM-1 Edge

Troubleshooting UTM-1 Edge

Debugging Sofaware Management Server
Edit $FWDIR/conf/sofaware/SWManagement.ini Change in line containing LogPolicy1 the value Info to Debug Smsstop sms confdir $FWDIR/conf/sofaware

Replicate the problem and watch for console output. Terminate programm and restart SMS afterwards
smsstart

Troubleshooting UTM-1 Edge

Configuration for Edge Devices on SPLAT under /opt/CPEdgecmp-R71/tmp <name <name <name <name <name of of of of of Edge Edge Edge Edge Edge object>.pf ruleset object>.pfz compressed ruleset object>.topo topology for VPN object>.tpz compressed topology object>.p12 PKCS#12 certificate

Delete files. Install policy again to re-generate them. Make sure, that the files are compiled and the Edge gets the latest version.

Opening a service request

Submit info to Check Point TAC or your CCSP/CSP provide contact info describe Check Point environment list used gateway hardware provide info about network topology and hardware describe the problem / the symptoms in detail what kind of business impact has this problem recommendation: get your supporter on the phone and be available for remote sessions use chat tool!

Opening a service request

Create compressed CPInfo diagnostic file
/opt/CPinfo-10/bin/cpinfo z <filename>

Create compressed CPInfo diagnostic file including logs

/opt/CPinfo-10/bin/cpinfo l z <filename>

CPInfo files can be viewed using InfoView

Make sure to have the latest CPinfo build installed! Check sk30567 for instructions!

TAC organisation
Director TAC Customer Focus Programmers

INTL Support

Escalations

Diamond Services

Knowledge Center

Data Security Escalation

3 Product Teams High end Core VPN

3 Product Teams High end Core VPN

Secure Knowledge

Technical Publications

TAC escalation
Support desk

Product team

Escalations

Customer focus programmer

TAC escalation path

http://www.checkpoint.com/services/contact/escalation.html

General debugging
kernel mode
usbcore rtmmod simmod vpntmod vpnmod fwmod

user mode
security server sms cpd fwd fwm

fw, VPN, FG-1, H323, BOA, WS, CPAS, CLUSTER, RTM, kiss, kissflow, multik, SFT, CI

kernel mode debug

View kernel modules with fw ctl debug h
kiss kissflow fw h323 multik BOA WS CI CPAS VPN RTM SFT Cluster FG-1 ?????? ??????? "Firewall Module" "VoIP H.323 Module" "related to CoreXL" "Malicious Code Protection Module" "SmartDefense Web Intelligence Module" Content Inspection "Active Streaming Module" "VPN Module" "SmartView Monitor Module" ??????? "ClusterXL Module" "Floodgate-1 QoS Module"

kernel mode debug

Some examples for modules and options: Module: fw Options: error warning cookie crypt domain ex driver filter hold if install ioctl kbuf ld log machine memory misc packet q xlate xltrc conn synatk media align balance chain bridge tcpstr scv ndis packval sync ipopt link nat cifs drop Module: vpn Options: driver err packet policy sas rdp clear cipher init sr comp xl counters mspi cphwd ref vin cluster nat l2tp warn

kernel mode debug

fw ctl debug Allocation of a buffer for the debug logs fw ctl debug buf [size in kb] The main debug command fw ctl debug m <module> <option> Writing the debug logs into a file fw ctl kdebug T f o <filename> Stop debugging fw ctl debug 0

kernel mode debug

Filter debug, only lines with <strings> in it are written to the output (best practice: error, failed) fw ctl debug d <strings> Filter debug, only lines that dont contain <string> in it are written to the output fw ctl debug d ^<strings> Can be combined fw ctl debug d error,failed,^packet

kernel mode debug

Stop debug messages when a certain string is issued. fw ctl debug s <string> Example: fw ctl debug s error

kernel mode debug

Example: debugging ClusterXL fw ctl fw ctl sync fw ctl fw ctl debug buf 32000 debug m fw + conn drop packet if debug m cluster all kdebug T f o <filename>

Example: debugging Site to Site VPN fw ctl debug -buf 32000 fw ctl debug -m VPN all fw ctl debug -m fw + conn drop ld xlate xltrc nat fw ctl kdebug T f o <filename>

kernel mode debug

Example: debugging SIP fw ctl debug buf 32000 fw ctl debug m fw + conn drop vm sip fw ctl kdebug T f o <filename> Example: debugging VoIP fw fw fw fw ctl ctl ctl ctl debug -buf 32000 debug -m fw + conn drop vm debug m h323 all kdebug T f o <filename>

kernel mode debug

Example: debugging SmartDefense fw ctl debug buf 32000 fw ctl debug m fw + conn drop vm tcp-str spii fw ctl kdebug T f o <filename> Example: debugging NAT fw ctl debug -buf 32000 fw ctl debug -m fw + xlate xltrc fw ctl kdebug T f o <filename>

kernel mode debug

Example: debugging QoS fw ctl debug buf 32000 fw ctl debug m FG-1 all fw ctl kdebug T f o <filename> Example: debugging SmartView Monitor fw ctl debug -buf 32000 fw ctl debug -m RTM all fw ctl kdebug T f o <filename>

VPN debug
Best practice before starting debug Compare configuration on both ends
often Phase I / Phase II parameters are not equal which causes the VPN to fail take special notice of networks and subnet masks carefully compare Pre-Shared-Secrets

Have a close look at the logs in SmartView Tracker Most informations can be found in the logs

VPN debug
To determine status of VPN tunnels, use menu based
vpn tunnelutil vpn tu

or SmartView Monitor To shutdown all VPN operation, use

vpn drv off

To enable VPN again, use

vpn drv on install policy

VPN debug
VPN debugging events can be logged on the gateway
vpn debug on Debug output is written to $FWDIR/log/vpnd.elg

More details can be logged using the command

vpn debug on TDERROR_ALL_ALL=5

Turn off debugging with

vpn debug off

VPN debug
IKE negotiations during VPN tunnel establishment can be logged in ike.elg On the gateway:
vpn debug ikeon / vpn debug ikeoff Debug output is written to $FWDIR/log/ike.elg

VPN debug
Initiate VPN and IKE debug together
vpn debug trunc

Disable VPN and IKE debug

vpn debug off vpn debug ikeoff

VPN debug
Capture traffic using fw monitor
fw monitor e accept port(500) or port(4500); o monitor.out Output file is monitor.out, IKE payloads are encrypted.

Capture traffic using vpn debug

vpn debug mon Output file is ikemonitor.snoop, IKE payloads are in clear. Turn off with vpn debug moff.

VPN debug
On UTM-1 Edge appliance:
WebUI -> Reports -> Tunnels -> save IKE trace

Click Save IKE Trace, which creates ike.elg

user mode debug

General syntax

fw debug <process> <on|off> TDERROR_ALL_ALL=<value> fw debug <process> <on|off> OPSEC_DEBUG_LEVEL=<value>

Exception: cpd

fwm debug
FWM controls connections from the SmartConsole to the SmartCenter server and is responsible for policy related functions To debug fwm do the following fw debug fwm on TDERROR_ALL_ALL=5 fw debug fwm on OPSEC_DEBUG_LEVEL=9 To stop debug run fw debug fwm off TDERROR_ALL_ALL=0 fw debug fwm off OPSEC_DEBUG_LEVEL=0 Logs are written to $FWDIR/log/fwm.elg

fwm debug
[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32] fwnetobj_getbysicname: table_chosen_get_with_param(eTABLE_NETWORK_OBJECTS, is_obj_SIC_name, IP=212.1.56.233,CN=Gui_Client) returned NULL. Login failed: 212.1.56.233 is not allowed for remote login [FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32] fwm_log: Login failed from IP=212.1.56.233,CN=Gui_Client: Unauthorized client Wed Sep 8 18:46:32 2010 (GMT): reject client IP=212.1.56.233,CN=Gui_Client [FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32] PM_policy_query: rule not found. [FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32] PM_policy_query: finished successfully. 1st method = deny IP not defined in $FWDIR/conf/gui-clients

fwm debug
[FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07] fwm_cpmi_auth_handler: authenticating admin admin by Name and Password [FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07] Administrator admin found in fwm database [FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07] CBinObjCommon::PackLogData: Field number:12, Data offset:34, Type:eFtCstring, Value:Administrator failed to log in: Wrong Password

fwd debug
FWD daemon controls logging, alerts, communication with the kernel, OPSEC communication, invokes child processes (security servers, ICA) To debug fwd do the following fw debug fwd on TDERROR_ALL_ALL=5 To stop debug run fw debug fwd off TDERROR_ALL_ALL=0

Logs are written to $FWDIR/log/fwd.elg

Desktop log server debug

To debug dtls do the following fw debug dtls on To stop debug run fw debug dtls off

Logs are written to $FWDIR/log/dtlsd.elg

Security servers debug

Some examples for security servers: FTP security server in.aftpd Telnet security server in.atelnetd HTTP security server in.ahttpd SMTP security server in.asmtpd ClientAuth (900) in.ahclientd ClientAuth (259) in.aclientd AntiSpam security server in.msd URL filtering security server in.aufpd

Security servers debug

Verify that security server process exists. Check $FWDIR/tmp for existing PID files. Start debugging (example for FTP security server) fw debug in.aftpd on FWAFTPD_LEVEL=3 Stop debugging fw debug in.aftpd off FWAFTPD_LEVEL=3

cpd debug
CPD controls SIC, Policy install To debug cpd do the following cpd_admin debug on TDERROR_ALL_ALL=5 To stop debug run cpd_admin debug off TDERROR_ALL_ALL=0

Logs are written to $CPDIR/log/cpd.elg

Secure Platform debug

Sometimes it is useful to verify file integrity and version against a test environment, for example after installation of ad-hoc fixes or HFA. Use md5sum for creating hashes.
[Expert@fwm]# md5sum upgrade_import e6c6417cca9db098b94673dd420a4903 upgrade_import

Use cpvinfo for displaying version information.

[Expert@fwm]# cpvinfo upgrade_import Build Number = 730080036 Major Release = NGX Minor Release = fli_up_ga Release Number = 5.0.5 Version Name = NGX

Secure Platform debug

For some problems with processes a core dump can be usefull. A core dump is a disk file that contains an image of the processs memory at the time of termination. Core dumps are mainly used by Check Point R&D for fixing a specific problem. Handling Core Files
http://downloads.checkpoint.com/dc/download.htm?ID=10479

Secure Platform debug

To enable core dumps do the following ulimit c unlimited um_core enable Reboot Check that /etc/sysconfig/enable_cores exist after Reboot. Dumps will be in /var/log/dump/usermode

Debugging GUI clients

Debug GUI clients Dashboard fwpolicy.exe d o fwp_debug.txt Tracker cplgv.exe d o cplgv_debug.txt Monitor smartcons.exe d o smartcons_debug.txt
general syntax: <executable> -d o <file_name>

Output is in specified directory or in

C:\Programme\CheckPoint\SmartConsole\R70\PROGRAM\data

if directory is omitted.

Resources
SmartSPLAT from ada Ulucan

www.smartsplat.com

Resources

Resources

Resources

Resources

Resources

Resources
fw monitor
http://www.checkpoint.com/techsupport/downloads/html/ ethereal/fw_monitor_rev1_01.pdf

The CPinfo utility

https://supportcenter.checkpoint.com/supportcenter/porta l?eventSubmit_doGoviewsolutiondetails=&solutionid=sk30 567

Documents related to troubleshooting

http://blog.lachmann.org/2010/09/documents-relatedto-troubleshooting/

Questions?

Still got a question?

Tobias Lachmann
[email protected] http://blog.lachmann.org