srx troubleshooting command library - juniper …troubleshooting*library* * page3*...
TRANSCRIPT
SRX Troubleshooting Library Compilation of Juniper SRX Troubleshooting Configurations and Commands
Ben Boyd Network Engineer
www.sinatranetwork.com
SRX Troubleshooting Library Page 2
Table of Contents
Copyright ........................................................................................................................................................... 5
Acknowledgements and Thanks ............................................................................................................... 5
Configuration Mode ....................................................................................................................................... 6 Verify the Last Committed Configuration ........................................................................................................... 6 show configuration ...................................................................................................................................................................... 6 show system commit .................................................................................................................................................................. 6 show configuration | compare rollback x .......................................................................................................................... 7 show configuration | display set ............................................................................................................................................ 7
Verify Logs Are Built .................................................................................................................................................. 8 messages log configuration ...................................................................................................................................................... 8 interactive-‐commands log configuration ........................................................................................................................... 8 blocked-‐traffic log configuration ........................................................................................................................................... 8 security log configuration ......................................................................................................................................................... 9
Verify Traceoptions Are Built .............................................................................................................................. 10 security flow traceoptions ...................................................................................................................................................... 10 ospf traceoptions ........................................................................................................................................................................ 10
Operational Mode ......................................................................................................................................... 11 Log Commands .......................................................................................................................................................... 11 show log messages ..................................................................................................................................................................... 11 show log interactive-‐commands .......................................................................................................................................... 11 show log jsrpd .............................................................................................................................................................................. 12 show log chassisd ....................................................................................................................................................................... 12 show system boot-‐messages ................................................................................................................................................. 12 monitor (start|stop) xyz .......................................................................................................................................................... 13 clear log xyz .................................................................................................................................................................................. 13 show log examples ..................................................................................................................................................................... 13
Alarm Commands .................................................................................................................................................... 14 show chassis alarms .................................................................................................................................................................. 14 show system alarms .................................................................................................................................................................. 14 show system core-‐dumps ....................................................................................................................................................... 14
Hardware Commands ............................................................................................................................................. 15 show chassis hardware detail ............................................................................................................................................... 15 show chassis environment ..................................................................................................................................................... 15 show chassis fan ......................................................................................................................................................................... 16
Software & Firmware Commands ...................................................................................................................... 17 show version ................................................................................................................................................................................ 17 show chassis firmware ............................................................................................................................................................. 17 show system software detail ................................................................................................................................................. 17
Usage Statistics Commands .................................................................................................................................. 19 show chassis routing-‐engine ................................................................................................................................................. 19 show system uptime ................................................................................................................................................................. 19 show system buffers ................................................................................................................................................................. 20
SRX Troubleshooting Library Page 3
show system virtual-‐memory ............................................................................................................................................... 20 show system processes ............................................................................................................................................................ 20 show security idp memory ..................................................................................................................................................... 21 show security monitoring performance session .......................................................................................................... 21 show security monitoring performance spu .................................................................................................................. 22 show security monitoring fpc X ........................................................................................................................................... 22
Cluster Commands .................................................................................................................................................. 23 show chassis cluster status .................................................................................................................................................... 23 show chassis cluster interfaces ............................................................................................................................................ 23 show chassis cluster statistics .............................................................................................................................................. 24 show chassis cluster information ........................................................................................................................................ 24
Interface Commands ............................................................................................................................................... 26 show interfaces terse | match reth ..................................................................................................................................... 26 show interfaces terse | match inet ...................................................................................................................................... 26 show interfaces ww-‐X/Y/Z | match zone ......................................................................................................................... 26 show interfaces ww-‐X/Y/Z extensive ............................................................................................................................... 26 monitor interface ww-‐X/Y/Z ................................................................................................................................................ 28 monitor traffic interface ww-‐X/Y/Z ................................................................................................................................... 28 monitor interface traffic .......................................................................................................................................................... 29
Routing Commands ................................................................................................................................................. 30 show ospf neighbor (instance xyz) ..................................................................................................................................... 30 show ospf database (instance xyz) ..................................................................................................................................... 30 show ospf route (instance xyz) ............................................................................................................................................ 30 show ospf statistics (instance xyz) ..................................................................................................................................... 31 show route [prefix] (table xyz) detail ................................................................................................................................ 31 show route protocol (ospf |bgp|static) ............................................................................................................................. 31 ping [destination] (routing-‐instance xyz) ........................................................................................................................ 32 traceroute [destination] (routing-‐instance xyz) (rapid) (count x) (size y) ...................................................... 32
Security Commands ................................................................................................................................................ 33 show security zones detail ..................................................................................................................................................... 33 show security flow statistics ................................................................................................................................................. 33 show security flow session summary ................................................................................................................................ 34 show security flow session (application|destination-‐prefix|source-‐prefix|…) ............................................... 34 show security alg status .......................................................................................................................................................... 35 show security nat source rule all ......................................................................................................................................... 36 show security nat destination rule all ............................................................................................................................... 36 show security nat static rule all ........................................................................................................................................... 36 show security policies (from-‐zone|policy-‐name|to-‐zone) ....................................................................................... 37
Contacting JTAC To Open A Technical Support Case .................................................................................... 38 Case Opening Procedure ......................................................................................................................................................... 38 request support information | save rsi_[date].txt ........................................................................................................ 38
Action Commands .................................................................................................................................................... 39 set chassis cluster cluster-‐id 1 node 1 reboot ................................................................................................................ 39 request chassis cluster failover redundancy-‐group 1 node 1 ................................................................................. 39 request chassis cluster failover reset redundancy-‐group 1 ..................................................................................... 39
SRX Troubleshooting Library Page 4
request system reboot .............................................................................................................................................................. 39 request system halt (request system power-‐off) ......................................................................................................... 39 request routing-‐engine login node 1 .................................................................................................................................. 39 request chassis pic fpc-‐slot 0 pic-‐slot 0 offline .............................................................................................................. 39 request system software add (location of image) no-‐validate no-‐copy reboot .............................................. 39
SRX Troubleshooting Library Page 5
Copyright This document is free for everyone. I just ask that you give credit where credit is due!
Acknowledgements and Thanks My awesome wife: Amanda.
My bosses past and present: Rachelle Summers, Joe Soricelli, Doug Marshke, and John Hasty.
The Juniper J-‐NET forum community and the Juniper twitter community.
SRX Troubleshooting Library Page 6
Configuration Mode Troubleshooting begins with configuration. The most common mistake when troubleshooting is not verifying the configuration is correct before racing to diagnose the issue. Operational mode commands are fantastic in helping diagnose and pinpoint problems, but in the end a configuration change will most likely “fix” the issue.
Verify the Last Committed Configuration You can verify the last committed configuration without entering configuration. Here are a few commands to help verify the configuration.
If there have been changes in the portion of configuration that is related to the issue you’re troubleshooting, verifying the configuration starts you in the right place. If there haven’t been any changes recently and the configuration looks correct, then you know you’re dealing with a possible hardware issue or something not related to the SRX at all.
show configuration This operational-‐mode command will show you the current running configuration as well as who committed this configuration.
## Last commit: 2010-‐09-‐09 08:26:46 UTC by ben version 10.0R3.10; system { host-‐name olive100; root-‐authentication { encrypted-‐password "$1$oafr8h7n$8h2yOCgqdtl7AIZHjloOh1"; ## SECRET-‐DATA } name-‐server { 208.67.222.222; } …
show system commit This operational-‐mode command shows the previously committed configuration, users who committed, and a timestamp of the commit. ben@olive100> show system commit 0 2010-‐09-‐09 08:26:46 UTC by ben via cli 1 2010-‐09-‐09 08:26:16 UTC by ben via cli 2 2010-‐09-‐06 09:03:52 UTC by ben via cli …
SRX Troubleshooting Library Page 7
show configuration | compare rollback x This operational-‐mode command compares the current configuration with a previously committed configuration (x). You can get the configuration number (x) from the ‘show system commit’ command.
ben@olive100> show configuration | compare rollback 2 [edit] + security { + flow { + inactive: traceoptions { + file flow_trace size 5m files 20 world-‐readable; + flag basic-‐datapath; + packet-‐filter to { + source-‐prefix 1.1.1.1/32; + destination-‐prefix 2.2.2.2/32; + } …
show configuration | display set This operational-‐mode command will show the configuration in “set” format. This helps with copying, editing, and pasting certain commands into the config.
ben@olive100> show configuration | display set set version 10.0R3.10 set system host-‐name olive100 set system root-‐authentication encrypted-‐password "$1$oafr8h7n$8h2yOCgqdtl7AIZHjloOh1" set system name-‐server 208.67.222.222 …
SRX Troubleshooting Library Page 8
Verify Logs Are Built The next step in the troubleshooting process is to verify that the SRX is correctly set up to log on event failures and issues.
messages log configuration Verify that the syslog is logging the system-‐wide messages you need. The messages log is the default JUNOS log for system-‐wide errors, alarms, and information.
ben@olive100> show configuration system syslog file messages any notice; authorization info;
interactive-‐commands log configuration The interactive-‐commands log is a custom log, but very useful when debugging what commands a user ran before the issue arose.
ben@olive100> show configuration system syslog file interactive-‐commands interactive-‐commands any;
blocked-‐traffic log configuration If you don’t have access or are unable to log from SRX security policies to a log server, the custom “blocked-‐traffic” log is great for logging policy denies. The log will not populate unless the security policy is set to log either session-‐init or session-‐close.
ben@olive100> show configuration system syslog file blocked-‐traffic any any; match RT_FLOW_SESSION_DENY; structured-‐data;
SRX Troubleshooting Library Page 9
security log configuration If you aren’t logging traffic policy denies/permits, troubleshooting policy issues can be extremely difficult. Below is a configuration for a security log stream. This configuration sends security policy logs to an external host.
ben@olive100> show configuration security log mode stream; format sd-‐syslog; source-‐address 10.203.234.2; stream STRM { severity info; format sd-‐syslog; category all; host { 10.203.234.4; port 514; } }
SRX Troubleshooting Library Page 10
Verify Traceoptions Are Built Traceoptions are available in virtually every portion of the JUNOS configuration. Since JUNOS runs processes in protected memory space, it is possible to trace (debug) individual configuration modules without affecting overall system performance.
security flow traceoptions The security flow traceoptions configuration is used to create a traceoptions file that debugs the flow of a packet matching a filter through the JUNOS flow processing module.
ben@olive100> show configuration security flow traceoptions file flow_trace size 5m files 20 world-‐readable; flag basic-‐datapath; packet-‐filter to { source-‐prefix 1.1.1.1/32; destination-‐prefix 2.2.2.2/32; } packet-‐filter from { source-‐prefix 2.2.2.2/32; destination-‐prefix 1.1.1.1/32; }
ospf traceoptions If OSPF is flapping or not exactly working right and you want more information than what is shown in the messages log (OSPF is down), then create a ospf specific traceoptions that captures the details of the OSPF operation.
ben@olive100> show configuration protocols ospf traceoptions file ospf_trace size 3m files 10 world-‐readable; flag all; flag state; flag spf; flag timer; flag task;
SRX Troubleshooting Library Page 11
Operational Mode Getting into the “meat” of troubleshooting and delving deep into JUNOS configuration, architecture, and processing is done through operational mode commands. Some of these commands are based on configurations we’ve built and some are built into JUNOS as a default.
This library doesn’t include every command, but it does include the bulk of operational troubleshooting commands you’ll need when encountering issues in your network. As with most network operating systems, navigating commands with the “?” key is extremely helpful.
Log Commands JUNOS logs are very helpful if they are configured correctly (see “Configurations” section above). This section shows how to view each of the relevant logs when dealing with issues within an SRX.
show log messages The messages log is the generic (not detailed) log for all events, errors, and information generated in the SRX.
ben@olive100> show log messages Sep 10 04:00:00 olive100 newsyslog[17631]: logfile turned over due to size>1024K Sep 10 04:00:06 olive100 /kernel: Process (14175,pkid) attempted to exceed RLIMIT_DATA: attempted 131136 KB Max 131072 KB Sep 10 04:05:06 olive100 /kernel: Process (14175,pkid) attempted to exceed RLIMIT_DATA: attempted 131136 KB Max 131072 KB Sep 10 04:10:06 olive100 /kernel: Process (14175,pkid) attempted to exceed RLIMIT_DATA: attempted 131136 KB Max 131072 KB Sep 10 04:15:06 …
show log interactive-‐commands This log is custom built (see configuration section) and extremely useful when finding out what users sent which commands to the SRX.
ben@olive100> show log interactive-‐commands Sep 4 17:00:00 olive100 newsyslog[14730]: logfile turned over due to size>1024K Sep 4 17:41:01 olive100 mgd[14422]: UI_CMDLINE_READ_LINE: User 'ben', command 'rollback 0 ' Sep 4 17:41:01 olive100 mgd[14422]: UI_LOAD_EVENT: User 'ben' is performing a 'rollback' Sep 4 17:41:02 olive100 mgd[14422]: UI_CMDLINE_READ_LINE: User 'ben', command 'exit ' …
SRX Troubleshooting Library Page 12
show log jsrpd The jsrpd log is the log generated by the jsrpd process that handles the SRX clustering. This log is relevant when troubleshooting cluster issues.
ben@olive100> show log jsrpd Nov 17 15:10:53 successfully set default traceoptions cfg Nov 17 15:10:53 JSRPD release 10.1R1.8 built by builder on 2010-‐02-‐12 17:29:39 UTC starting, pid 1110 Nov 17 15:10:53 node id invalid, cluster-‐id 0 in kernel Nov 17 15:10:53 Control interface name em0 with index 0 …
show log chassisd The chassisd log is the log generated by the chassisd process that handles the Juniper SRX chassis environment. If you have a card go up in flames, you’ll see the detailed alarms and messages in this log.
ben@olive100> show log chassisd Dec 9 19:52:41 ge-‐1/0/6: large delay buffer cleared Dec 9 19:52:41 ge-‐1/0/6: ingress queueing cleared for QDPC Dec 9 19:52:41 CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for ge-‐1/0/7 …
show system boot-‐messages This “log” contains the messages produced during the boot sequence of the device.
ben@olive100> show system boot-‐messages Copyright (c) 1996-‐2010, Juniper Networks, Inc. All rights reserved. Copyright (c) 1992-‐2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. JUNOS 10.0R3.10 #0: 2010-‐04-‐16 07:17:53 UTC [email protected]:/volume/build/junos/10.0/release/10.0R3.10/obj-‐i386/bsd/sys/compile/JSR Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU P7550 @ 2.26GHz (2257.38-‐MHz 686-‐class CPU) …
SRX Troubleshooting Library Page 13
monitor (start|stop) xyz The “monitor” command will start or stop the LIVE streaming of a log to the terminal. If you monitor a very active log, your terminal screen will be overrun with log messages, so be careful.
ben@olive100> monitor start messages
clear log xyz If you have a large log and have gone through as much as you can without finding anything relevant, you can clear a log to start fresh.
ben@olive100> clear log interactive-‐commands
show log examples Here are some examples for showing logs.
ben@sinatra-‐fw1-‐node0> show log messages | match alarm Feb 23 14:32:53 sinatra-‐fw1-‐node0 craftd[1157]: Minor alarm set, Host 0 Temperature Warm Feb 23 14:32:58 sinatra-‐fw1-‐node0 alarmd[1108]: Alarm cleared: RE color=YELLOW, class=CHASSIS, reason=Host 0 Temperature Warm Feb 23 14:32:58 sinatra-‐fw1-‐node0 craftd[1157]: Minor alarm cleared, Host 0 Temperature Warm
ben@sinatra-‐fw1-‐node0> show log interactive-‐commands | last 5 Feb 24 14:05:43 sinatra-‐fw1-‐node0 mgd[7314]: UI_CMDLINE_READ_LINE: User 'ben', command 'show log messages | match alarm ' Feb 24 14:06:51 sinatra-‐fw1-‐node0 mgd[7314]: UI_CMDLINE_READ_LINE: User 'ben', command 'show log interactive-‐commands | last 5 '
ben@sinatra-‐fw1-‐node0> show log messages | find 14:05 Feb 24 14:05:37 sinatra-‐fw1-‐node0 sshd[7310]: Accepted password for ben from 10.0.100.3 port 57952 ssh2
SRX Troubleshooting Library Page 14
Alarm Commands JUNOS creates alarms when the environment is not operating as manufactured/configured. Below are the commands to view those alarms.
show chassis alarms This command shows alarms solely related to the hardware and the chassis.
ben@olive100> show chassis alarms 2 alarms currently active Alarm time Class Description 2010-‐08-‐27 21:24:52 UTC Major Jseries Chassis fan Failure 2010-‐08-‐27 21:24:52 UTC Major Jseries CPU fan Failure
show system alarms This command shows alarms throughout the system, which can include chassis alarms as well.
ben@olive100> show system alarms 3 alarms currently active Alarm time Class Description 2010-‐08-‐27 21:24:52 UTC Major Jseries Chassis fan Failure 2010-‐08-‐27 21:24:52 UTC Major Jseries CPU fan Failure 2010-‐08-‐27 21:24:19 UTC Minor Rescue configuration is not set
show system core-‐dumps If a CPU fails on any of the hardware installed, the CPU dumps the core contents into a file. These files are saved on the routing-‐engine’s hard drive for review by JTAC technicians.
ben@olive100> show system core-‐dumps /var/crash/*core*: No such file or directory -‐rw-‐rw-‐-‐-‐-‐ 1 root wheel 654693 Sep 4 03:16 /var/tmp/flowd_hm.core.0.gz -‐rw-‐rw-‐-‐-‐-‐ 1 root wheel 654696 Sep 4 03:16 /var/tmp/flowd_hm.core.1.gz -‐rw-‐rw-‐-‐-‐-‐ 1 root wheel 654693 Sep 4 03:16 /var/tmp/flowd_hm.core.2.gz /var/crash/kernel.*: No such file or directory /tftpboot/corefiles/*core*: No such file or directory total 3
SRX Troubleshooting Library Page 15
Hardware Commands If you are troubleshooting what you believe to be hardware issues, the following commands will be useful in determining the hardware environment of the SRX.
show chassis hardware detail This command shows all of the installed hardware in the device. It would be best to have a baseline of this command before issues are ran into, sometimes when hardware goes bad it won’t show in this output. This command is also used to find the serial number of the device when opening a JTAC case.
juniper@cascrmdinet50rd-‐f1> show chassis hardware detail node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Hardware inventory: Item Version Part number Serial number Description Chassis JN11A270DAGA SRX 5800 Midplane REV 01 710-‐024803 ABAB4976 SRX 5800 Backplane FPM Board REV 01 710-‐024632 YG4935 Front Panel Display PDM Rev 03 740-‐013110 QCS142350CF Power Distribution Module PEM 0 Rev 03 740-‐023514 QCS1401E00H PS 1.7kW; 200-‐240VAC in … node1: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Hardware inventory: Item Version Part number Serial number Description Chassis JN11A2706AGA SRX 5800 Midplane REV 01 710-‐024803 ABAB4980 SRX 5800 Backplane FPM Board REV 01 710-‐024632 YF9526 Front Panel Display PDM Rev 03 740-‐013110 QCS142350BP Power Distribution Module PEM 0 Rev 03 740-‐023514 QCS1435E00W PS 1.7kW; 200-‐240VAC in …
show chassis environment This command shows the temperature and status of all hardware components in the SRX.
juniper@cascrmdinet50rd-‐f1> show chassis environment node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Class Item Status Measurement Temp PEM 0 OK 40 degrees C / 104 degrees F PEM 2 OK 35 degrees C / 95 degrees F
SRX Troubleshooting Library Page 16
Bottom Tray Fan 6 OK Spinning at normal speed … node1: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Class Item Status Measurement Temp PEM 0 OK 40 degrees C / 104 degrees F PEM 2 OK 35 degrees C / 95 degrees F Bottom Tray Fan 6 OK Spinning at normal speed …
show chassis fan This command shows the status of all of the fans in the SRX (also part of the command above)
juniper@cascrmdinet50rd-‐f1> show chassis fan node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Item Status RPM Measurement Top Tray Fan 1 OK 2896 Spinning at normal speed … node1: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Item Status RPM Measurement Top Tray Fan 1 OK 2880 Spinning at normal speed …
SRX Troubleshooting Library Page 17
Software & Firmware Commands As everyone knows, software and firmware versions make all of the difference. The commands below help verify the versions of software and firmware on the SRX.
show version This command shows the version of JUNOS loaded on the SRX.
ben@olive100> show version Hostname: olive100 Model: j4300 JUNOS Software Release [10.0R3.10]
show chassis firmware This command shows the firmware version loaded on each FPC.
juniper@cascrmdinet50rd-‐f1> show chassis firmware node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Part Type Version FPC 1 ROM Juniper ROM Monitor Version 9.5b1 O/S Version 10.2R3.10 by builder on 2010-‐10-‐16 FPC 9 ROM Juniper ROM Monitor Version 9.5b1 O/S Version 10.2R3.10 by builder on 2010-‐10-‐16 FPC 10 ROM Juniper ROM Monitor Version 9.5b1 O/S Version 10.2R3.10 by builder on 2010-‐10-‐16 … node1: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Part Type Version FPC 1 ROM Juniper ROM Monitor Version 9.5b1 O/S Version 10.2R3.10 by builder on 2010-‐10-‐16 FPC 9 ROM Juniper ROM Monitor Version 9.5b1 O/S Version 10.2R3.10 by builder on 2010-‐10-‐16 FPC 10 ROM Juniper ROM Monitor Version 9.5b1 O/S Version 10.2R3.10 by builder on 2010-‐10-‐16 …
show system software detail This is a more detailed “show version” command.
SRX Troubleshooting Library Page 18
juniper@cascrmdinet50rd-‐f1> show system software detail node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Information for junos: Comment: JUNOS Software Release [10.2R3.10] Depends on: Description: JUNOS Software Release Copyright (c) 1996-‐2010, Juniper Networks, Inc. All rights reserved. Software version: 10.2R3.10 This package contains OS components. …
SRX Troubleshooting Library Page 19
Usage Statistics Commands As with any device with memory and a CPU, you want to check those vital signs on the SRX as well.
Typical customer thresholds for alarms:
Routing-‐engine CPU Usage: 60% Routing-‐engine Memory Usage: 80% (depending on BGP status 90% may be acceptable) SPU CPU Usage: 60% IDP Memory Usage: 70%
show chassis routing-‐engine This command shows vitals such as CPU and Memory utilization for the Routing-‐Engine (routing brains).
ben@olive100> show chassis routing-‐engine Routing Engine status: Total memory 1024 MB Max 502 MB used ( 49 percent) Control plane memory 594 MB Max 499 MB used ( 84 percent) Data plane memory 430 MB Max 0 MB used ( 0 percent) CPU utilization: User 81 percent Real-‐time threads 0 percent Kernel 19 percent Idle 0 percent Start time 2010-‐08-‐27 21:23:43 UTC Uptime 13 days, 18 hours, 31 minutes, 58 seconds Last reboot reason 0x8:power-‐button hard power off Load averages: 1 minute 5 minute 15 minute 1.00 1.00 1.00
show system uptime This command lists the current total running time of the device.
ben@olive100> show system uptime Current time: 2010-‐09-‐10 15:55:19 UTC System booted: 2010-‐08-‐27 21:23:43 UTC (1w6d 18:31 ago) Protocols started: 2010-‐08-‐27 21:24:21 UTC (1w6d 18:30 ago) Last configured: 2010-‐09-‐10 14:02:18 UTC (01:53:01 ago) by ben 3:55PM up 13 days, 18:32, 1 user, load averages: 1.00, 1.00, 1.00
SRX Troubleshooting Library Page 20
show system buffers This command shows the current utilization of the various memory buffers within the SRX.
ben@olive100> show system buffers 1875/315/2190 mbufs in use (current/cache/total) 1539/147/1686/20640 mbuf clusters in use (current/cache/total/max) 1536/128 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 3546K/372K/3919K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/4/640 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines
show system virtual-‐memory This command shows the memory utilization of each process in the SRX.
ben@olive100> show system virtual-‐memory Type InUse MemUse HighUse Requests Size(s) ata_dma 2 1K -‐ 2 256 file desc 117 25K -‐ 25635 16,1024,2048,16384 proc-‐args 45 2K -‐ 16515 16,32,64,128,256,512,1024,2048,4096 … 849545997 cpu context switches 1494111802 device interrupts 78308832 software interrupts 5881305 traps 4257155619 system calls 50 kernel threads created …
show system processes This is the equivalent of the “ps” command in a UNIX environment. It shows all of the current running processes on the SRX.
ben@olive100> show system processes
SRX Troubleshooting Library Page 21
PID TT STAT TIME COMMAND 0 ?? WLs 0:00.00 [swapper] 1 ?? ILs 0:01.19 /junos/sbin/init -‐D/junos -‐-‐ 2 ?? DL 0:33.36 [g_event] …
show security idp memory This command shows the memory usage of the IDP process on each FPC.
juniper@cascrmdinet50rd-‐f1> show security idp memory node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ IDP data plane memory statistics: PIC : FPC 11 PIC 1: Total IDP data plane memory : 515 MB Used : 40 MB ( 40960 KB ) ( 7.77%) Available : 475 MB ( 486400 KB ) ( 92.23%) PIC : FPC 11 PIC 0: …
show security monitoring performance session This command shows the session counts on each FPC.
juniper@cascrmdinet50rd-‐f1> show security monitoring performance session node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ fpc 9 pic 1 Last 60 seconds: 0: 2412 1: 2360 2: 2419 3: 2350 4: 2433 5: 2379 6: 2431 7: 2369 8: 2434 9: 2373 10: 2436 11: 2375 12: 2423 13: 2361 14: 2409 15: 2350 16: 2415 17: 2358 18: 2409 19: 2344 20: 2404 21: 2346 22: 2439 23: 2381 24: 2465 25: 2400 26: 2464 27: 2402 28: 2476 29: 2405 30: 2483 31: 2426 32: 2495 33: 2425 34: 2462 35: 2400 36: 2480 37: 2418 38: 2569 39: 2513 40: 2571 41: 2509 42: 2575 43: 2518 44: 2578 45: 2519 46: 2561 47: 2506 48: 2563 49: 2501 50: 2545 51: 2480 52: 2545 53: 2492 54: 2562 55: 2504 56: 2563 57: 2507 58: 2562 59: 2504 …
SRX Troubleshooting Library Page 22
show security monitoring performance spu This command shows the performance statistics for the spu on each FPC.
juniper@cascrmdinet50rd-‐f1> show security monitoring performance spu node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ fpc 11 pic 0 Last 60 seconds: 0: 2 1: 2 2: 3 3: 2 4: 2 5: 1 6: 2 7: 2 8: 3 9: 3 10: 2 11: 2 12: 2 13: 3 14: 4 15: 3 16: 3 17: 3 18: 3 19: 3 20: 3 21: 2 22: 2 23: 3 24: 3 25: 3 26: 2 27: 2 28: 3 29: 2 30: 3 31: 4 32: 4 33: 3 34: 2 35: 3 36: 3 37: 2 38: 2 39: 2 40: 2 41: 2 42: 3 43: 3 44: 3 45: 3 46: 3 47: 4 48: 3 49: 2 50: 2 51: 3 52: 2 53: 1 54: 2 55: 2 56: 2 57: 2 58: 3 59: 3 …
show security monitoring fpc X This command shows the performance statistics for the FPC selected.
juniper@cascrmdinet50rd-‐f1> show security monitoring fpc 9 | no-‐more node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ FPC 9 PIC 0 CPU utilization : 0 % Memory utilization : 81 % Current flow session : 0 Max flow session : 0 Current CP session : 11453 Max CP session : 10485760 PIC 1 CPU utilization : 0 % Memory utilization : 64 % Current flow session : 2369 Max flow session : 1048576 Current CP session : 0 Max CP session : 0
SRX Troubleshooting Library Page 23
Cluster Commands When troubleshooting issues in an SRX environment, one of the first areas you’ll need to verify as operational is the “clustering” of 2 physical nodes into 1 logical node. If the cluster is built or performing correctly many other system and network issues can creep up as a result.
show chassis cluster status This command shows the status of the SRX cluster. Conditions to watch for:
• Status other than “Primary” or “Secondary” • Priority other than what’s configured (typically 254 & 1) • Manual Failover other than “no”
juniper@cascrmdinet50rd-‐f1> show chassis cluster status Cluster ID: 1 Node Priority Status Preempt Manual failover Redundancy group: 0 , Failover count: 1 node0 254 primary no no node1 1 secondary no no Redundancy group: 1 , Failover count: 8 node0 254 secondary no no node1 1 primary no no
show chassis cluster interfaces This command shows all of the interfaces involved in the SRX cluster. This includes the control ports, the fabric ports, redundant Ethernet interfaces (reth), and the monitored network ports.
juniper@cascrmdinet50rd-‐f1> show chassis cluster interfaces Control link 0 name: em0 Control link 1 name: em1 Control link status: Up Fabric interfaces: Name Child-‐interface Status fab0 ge-‐1/0/15 up fab1 ge-‐13/0/15 up Fabric link status: Up Redundant-‐ethernet Information: Name Status Redundancy-‐group reth0 Down Not configured reth1 Up 1
SRX Troubleshooting Library Page 24
reth2 Up 1 Interface Monitoring: Interface Weight Status Redundancy-‐group ge-‐13/0/14 255 Up 1 …
show chassis cluster statistics This command shows the counters involved in the SRX cluster environment. The control link and fabric link sent and received counts should increment with the re-‐running of this command.
juniper@cascrmdinet50rd-‐f1> show chassis cluster statistics Control link statistics: Control link 0: Heartbeat packets sent: 1474309 Heartbeat packets received: 1473945 Heartbeat packet errors: 0 Control link 1: Heartbeat packets sent: 0 Heartbeat packets received: 0 Heartbeat packet errors: 0 Fabric link statistics: Probes sent: 1474291 Probes received: 1272362 Probe errors: 0 Services Synchronized: Service name RTOs sent RTOs received Translation context 0 0 Incoming NAT 0 0 Resource manager 0 0 Session create 0 181353670 …
show chassis cluster information This is a hidden command that combines much of the data already presented above along with other relevant cluster information.
juniper@cascrmdinet50rd-‐f1> show chassis cluster statistics Control link statistics: Control link 0: Heartbeat packets sent: 1474309 Heartbeat packets received: 1473945
SRX Troubleshooting Library Page 25
Heartbeat packet errors: 0 Control link 1: Heartbeat packets sent: 0 Heartbeat packets received: 0 Heartbeat packet errors: 0 Fabric link statistics: Probes sent: 1474291 Probes received: 1272362 Probe errors: 0 Services Synchronized: Service name RTOs sent RTOs received Translation context 0 0 Incoming NAT 0 0 Resource manager 0 0 Session create 0 181353670 …
SRX Troubleshooting Library Page 26
Interface Commands
show interfaces terse | match reth This command shows all interfaces associated with reth interfaces and their up/down admin and physical status
juniper@cascrmdinet50rd-‐f1> show interfaces terse | match reth ge-‐1/0/0.0 up up aenet -‐-‐> reth1.0 ge-‐13/0/0.0 up up aenet -‐-‐> reth1.0 reth0 up down reth1 up up reth1.0 up up inet 10.255.51.183/28 …
show interfaces terse | match inet This command shows all interfaces with IP addresses configured.
juniper@cascrmdinet50rd-‐f1> show interfaces terse | match inet em0.0 up up inet 129.16.0.1/2 em1.0 up up inet 129.16.0.1/2 reth1.0 up up inet 10.255.51.183/28 reth2.0 up up inet 10.255.51.167/28 reth10.0 up up inet 162.115.8.210/23
show interfaces ww-‐X/Y/Z | match zone This command shows the zone associated with the specificed interface.
juniper@cascrmdinet50rd-‐f1> show interfaces reth1 | match zone Security: Zone: red
show interfaces ww-‐X/Y/Z extensive This command shows extensive statistics and status for the specified interface. Some things to take note of: MTU, Speed, Flow Control, Device Flags, Current Address, Last Flapped, Input Errors, Output Errors, Flow Error Statistics (Especially TCP Sequence out of window) juniper@cascrmdinet50rd-‐f1> show interfaces reth1 extensive Physical interface: reth1, Enabled, Physical link is Up Interface index: 129, SNMP ifIndex: 522, Generation: 132
SRX Troubleshooting Library Page 27
Link-‐level type: Ethernet, MTU: 1514, Speed: 1Gbps, BPDU Error: None, MAC-‐REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0 Device flags : Present Running Interface flags: SNMP-‐Traps Internal: 0x4000 Current address: 00:10:db:ff:10:01, Hardware address: 00:10:db:ff:10:01 Last flapped : 2010-‐12-‐12 22:00:41 GMT (1w2d 00:53 ago) Statistics last cleared: Never Traffic statistics: Input bytes : 787970088269 8851872 bps Output bytes : 8881734839165 95182056 bps Input packets: 4921133214 7158 pps Output packets: 7887317751 10800 pps Input errors: Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0 Output errors: Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0 … Security: Zone: red Allowed host-‐inbound traffic : ospf Flow Statistics : Flow Input statistics : Bytes permitted by policy : 629022590737 Connections established : 181147640 Flow Output statistics: Multicast packets : 0 Bytes permitted by policy : 8640720901899 Flow error statistics (Packets dropped due to): … No zone or NULL zone binding 0 Policy denied: 420979 Security association not active: 0 TCP sequence number out of window: 311511 Protocol inet, MTU: 1500, Generation: 153, Route table: 6 Addresses, Flags: Is-‐Default Is-‐Preferred Is-‐Primary Destination: 10.255.51.176/28, Local: 10.255.51.183, Broadcast: 10.255.51.191, Generation: 140 Protocol multiservice, MTU: Unlimited, Generation: 154, Route table: 6 …
SRX Troubleshooting Library Page 28
monitor interface ww-‐X/Y/Z This command shows live interface counts for input/output packets and errors. This is a LIVE command that requires you to exit out of the command (q).
cascrmdinet50rd-‐f1 Seconds: 4 Time: 23:01:15 Delay: 0/0/2 Interface: reth1, Enabled, Link is Up Encapsulation: Ethernet, Speed: 1000mbps Traffic statistics: Current delta Input bytes: 788500518244 (8939808 bps) [3711746] Output bytes: 8887432578672 (98848840 bps) [42499178] Input packets: 4924386289 (6933 pps) [23352] Output packets: 7892464225 (11443 pps) [38473] Error statistics: Input errors: 0 [0] Input drops: 0 [0] Input framing errors: 0 [0] Carrier transitions: 0 [0] Output errors: 0 [0] Output drops: 0 [0]
monitor traffic interface ww-‐X/Y/Z This command is a tcpdump of traffic destined for the interface specified. This is a LIVE command that requires you to exit out of the command (CTRL+C). Note: ping (ICMP) traffic and transit traffic will not appear in this display
juniper@cascrmdinet50rd-‐f1> monitor traffic interface reth1 Listening on reth1, capture size 96 bytes Reverse lookup for 10.255.51.183 failed (check DNS reachability). Other reverse lookup failures will not be reported. Use <no-‐resolve> to avoid reverse lookups on IP addresses. 23:02:04.599618 Out IP truncated-‐ip -‐ 12 bytes missing! 10.255.51.183 > OSPF-‐ALL.MCAST.NET: OSPFv2, Hello, length 52 23:02:05.570679 In IP 10.255.51.178 > OSPF-‐DSIG.MCAST.NET: OSPFv2, LS-‐Update, length 56 23:02:05.570691 In IP 10.255.51.179 > OSPF-‐ALL.MCAST.NET: OSPFv2, LS-‐Update, length 56 23:02:05.605954 In IP 10.255.51.178 > OSPF-‐DSIG.MCAST.NET: OSPFv2, LS-‐Update, length 56 …
SRX Troubleshooting Library Page 29
monitor interface traffic This command shows the input and output packet counts for all interfaces on the SRX Cluster. This is a LIVE command that requires you to exit out of the command (q).
cascrmdinet50rd-‐f1 Seconds: 6 Time: 23:03:48 Interface Link Input packets (pps) Output packets (pps) ge-‐1/0/0 Up 1787893 (0) 0 (0) ge-‐1/0/1 Down 0 (0) 0 (0) ge-‐1/0/2 Up 1760802 (0) 0 (0) ge-‐1/0/3 Down 0 (0) 0 (0) ge-‐1/0/14 Up 11340070 (14) 0 (0) ge-‐1/0/15 Up 0 (0) 4049328 (5) mt-‐9/0/0 Down 0 (0) 0 (0) ge-‐13/0/0 Up 4923779005 (7035) 7894549362 (10754) ge-‐13/0/1 Down 0 (0) 0 (0) ge-‐13/0/2 Up 7900541604 (10751) 4910745880 (7025) ge-‐13/0/3 Down 0 (0) 0 (0) ge-‐13/0/14 Up 11374760 (15) 167351779 (634) ge-‐13/0/15 Up 0 (0) 366188899 (654) mt-‐21/0/0 Down 0 (0) 0 (0) …
SRX Troubleshooting Library Page 30
Routing Commands
show ospf neighbor (instance xyz) This command shows the OSPF neighbors for a specific routing-‐instance
juniper@cascrmdinet50rd-‐f1> show ospf neighbor instance prod-‐vr Address Interface State ID Pri Dead 10.255.51.178 reth1.0 Full 10.255.63.5 10 38 10.255.51.179 reth1.0 Full 10.255.63.6 5 33 10.255.51.162 reth2.0 Full 10.255.63.11 10 37 10.255.51.163 reth2.0 Full 10.255.63.12 5 31
show ospf database (instance xyz) This command shows the OSPF database for a specific routing-‐instance
juniper@cascrmdinet50rd-‐f1> show ospf database instance prod-‐vr OSPF database, Area 0.0.0.0 Type ID Adv Rtr Seq Age Opt Cksum Len Router 10.254.64.46 10.254.64.46 0x80000940 155 0x2 0xd437 72 Router 10.254.64.47 10.254.64.47 0x80000940 157 0x2 0x17f0 72 Router 10.254.115.120 10.254.115.120 0x8000022b 861 0x2 0x5c6f 72 …
show ospf route (instance xyz) This command displays the route-‐table built by the OSPF SPF algorithm. This table is then inserted into the routing table for the routing-‐instance it belongs to.
juniper@cascrmdinet50rd-‐f1> show ospf route instance prod-‐vr Topology default Route Table: Prefix Path Route NH Metric NextHop Nexthop Type Type Type Interface Address/LSP 10.254.64.46 Intra AS BR IP 81 reth2.0 10.255.51.162 10.254.64.47 Intra Router IP 81 reth2.0 10.255.51.162 10.254.115.120 Intra Router IP 81 reth2.0 10.255.51.162 …
SRX Troubleshooting Library Page 31
show ospf statistics (instance xyz) This command shows counters for OSPF related traffic. This is useful in in determining if routes are leaving the OSPF process and reaching the routing-‐engine.
juniper@cascrmdinet50rd-‐f1> show ospf statistics instance prod-‐vr Packet type Total Last 5 seconds Sent Received Sent Received Hello 337689 605605 2 1 DbD 3995 3960 0 0 LSReq 125 2 0 0 LSUpdate 393445 1033018 0 0 …
show route [prefix] (table xyz) detail This command shows detailed information concerning the specified route prefix.
juniper@cascrmdinet50rd-‐f1> show route 0.0.0.0 table prod-‐vr detail prod-‐vr.inet.0: 2077 destinations, 2077 routes (2077 active, 0 holddown, 0 hidden) 0.0.0.0/0 (1 entry, 1 announced) *OSPF Preference: 150 Next hop type: Router, Next hop index: 599 Next-‐hop reference count: 2895 Next hop: 10.255.51.178 via reth1.0, selected State: <Active Int Ext> Age: 1w2d 1:13:45 Metric: 501 Tag: 1 Task: prod-‐vr-‐OSPF Announcement bits (1): 2-‐KRT AS path: I
show route protocol (ospf |bgp|static) This command will show the routing table, but only routes that match the specified protocol.
juniper@cascrmdinet50rd-‐f1> show route protocol static inet.0: 7 destinations, 7 routes (6 active, 0 holddown, 1 hidden) + = Active Route, -‐ = Last Active, * = Both
SRX Troubleshooting Library Page 32
0.0.0.0/0 *[Static/5] 2w3d 02:33:03 > to 162.115.8.1 via fxp0.0 162.115.9.31/32 *[Static/5] 2w3d 02:33:03 to table logging.inet.0 162.115.9.36/32 *[Static/5] 2w3d 02:33:03 to table logging.inet.0 162.115.9.221/32 *[Static/5] 2w3d 02:33:03 to table logging.inet.0 logging.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) + = Active Route, -‐ = Last Active, * = Both 0.0.0.0/0 *[Static/5] 1w2d 01:25:59 > to 162.115.8.1 via reth10.0 prod-‐vr.inet.0: 2077 destinations, 2077 routes (2077 active, 0 holddown, 0 hidden) + = Active Route, -‐ = Last Active, * = Both 162.115.40.1/32 *[Static/5] 1w2d 01:26:00 > to 10.255.51.178 via reth1.0
ping [destination] (routing-‐instance xyz) This command sends an ICMP ping from the SRX to the destination in the specified routing-‐instance.
{primary:node0} juniper@cascrmdinet50rd-‐f1> ping 10.255.51.178 routing-‐instance prod-‐vr PING 10.255.51.178 (10.255.51.178): 56 data bytes 64 bytes from 10.255.51.178: icmp_seq=0 ttl=255 time=2.092 ms 64 bytes from 10.255.51.178: icmp_seq=1 ttl=255 time=2.092 ms ^C -‐-‐-‐ 10.255.51.178 ping statistics -‐-‐-‐ 2 packets transmitted, 2 packets received, 0% packet loss round-‐trip min/avg/max/stddev = 2.092/2.092/2.092/0.000 ms
traceroute [destination] (routing-‐instance xyz) (rapid) (count x) (size y) This command starts an ICMP network trace-‐route to the destination in the specified routing-‐instance.
juniper@cascrmdinet50rd-‐f1> traceroute 10.255.51.178 routing-‐instance prod-‐vr traceroute to 10.255.51.178 (10.255.51.178), 30 hops max, 40 byte packets 1 10.255.51.178 (10.255.51.178) 2.447 ms * 1.948 ms
SRX Troubleshooting Library Page 33
Security Commands
show security zones detail This command shows all of the configured security zones on the SRX and the interfaces associated with them.
juniper@cascrmdinet50rd-‐f1> show security zones detail node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Security zone: logging Send reset for non-‐SYN session TCP packets: Off Policy configurable: Yes Interfaces bound: 1 Interfaces: reth10.0 Security zone: red Send reset for non-‐SYN session TCP packets: Off Policy configurable: Yes Interfaces bound: 1 Interfaces: reth1.0 Security zone: yellow Send reset for non-‐SYN session TCP packets: Off Policy configurable: Yes Interfaces bound: 1 Interfaces: reth2.0
show security flow statistics This command gives a flow statistics for each SPU.
juniper@cascrmdpcign-‐f1> show security flow statistics node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Flow Statistics of FPC9 PIC1: Current sessions: 4287 Packets forwarded: 0 Packets dropped: 102995758 Fragment packets: 0
SRX Troubleshooting Library Page 34
… Flow Statistics Summary: System total valid sessions: 21005 Packets forwarded: 0 Packets dropped: 439358642 Fragment packets: 0 …
show security flow session summary This command gives a total count of the sessions on each SPU.
juniper@cascrmdinet50rd-‐f1> show security flow session summary node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Flow Sessions on FPC9 PIC1: Unicast-‐sessions: 2362 Multicast-‐sessions: 0 Failed-‐sessions: 0 Sessions-‐in-‐use: 2581 Valid sessions: 2344 Pending sessions: 0 Invalidated sessions: 219 Sessions in other states: 0 Maximum-‐sessions: 1048576 …
show security flow session (application|destination-‐prefix|source-‐prefix|…) This command shows all sessions and session detials that match the specified parameters. Items to consider: Policy Name, Source NAT pool, Application, Session ID, In and Out addresses and ports, In and Out interfaces, and FIN states juniper@cascrmdinet50rd-‐f1> show security flow session session-‐identifier 370002887 Flow Sessions on FPC9 PIC1: Session ID: 370002887, Status: Normal, State: Backup Flag: 0x10000040 Policy name: 5/8 Source NAT pool: Null, Application: junos-‐https/58
SRX Troubleshooting Library Page 35
Maximum timeout: 1800, Current timeout: 24 Session State: Valid Start time: 781938, Duration: 1384 In: 66.38.121.149/1634 -‐-‐> 162.115.18.100/443;tcp, Interface: reth1.0, Session token: 0x18c, Flag: 0x0x2621 Route: 0x0, Gateway: 66.38.121.149, Tunnel: 0 Port sequence: 0, FIN sequence: 0, FIN state: 0, Pkts: 0, Bytes: 0 Out: 162.115.18.100/443 -‐-‐> 66.38.121.149/1634;tcp, Interface: reth2.0, Session token: 0x1cc, Flag: 0x0x2620 Route: 0x0, Gateway: 162.115.18.100, Tunnel: 0 Port sequence: 0, FIN sequence: 123069361, FIN state: 1, Pkts: 0, Bytes: 0 Total sessions: 1
show security alg status This command shows all possible application layer gateways on the SRX and their status. All ALGs are enabled by default.
juniper@cascrmdinet50rd-‐f1> show security alg status ALG Status : DNS : Enabled FTP : Enabled H323 : Disabled MGCP : Disabled MSRPC : Enabled PPTP : Enabled RSH : Enabled RTSP : Disabled SCCP : Disabled SIP : Disabled SQL : Enabled SUNRPC : Enabled TALK : Enabled TFTP : Enabled IKE-‐ESP : Disabled
SRX Troubleshooting Library Page 36
show security nat source rule all This command shows source NAT statistics and configuration information.
juniper@cascrmdinet50rd-‐f1> show security nat source rule all node1: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Total rules: 2 source NAT rule: 1 Rule-‐set: sdc-‐outbound-‐nat Rule-‐Id : 3 Rule position : 1 From zone : yellow To zone : red Match Source addresses : 10.255.9.0 -‐ 10.255.9.127 Destination addresses : 69.78.139.61 -‐ 69.78.139.61 96.6.134.98 -‐ 96.6.134.98 Destination port : 0 -‐ 0 Action : pool1 Persistent NAT type : N/A Persistent NAT mapping type : address-‐port-‐mapping Inactivity timeout : 0 Max session number : 0 Translation hits : 69518 …
show security nat destination rule all This command shows destination NAT statistics and configuration information.
juniper@cascrmdinet50rd-‐f1> show security nat destination rule all node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ Total destination-‐nat rules: 0 …
show security nat static rule all This command shows destination NAT statistics and configuration information.
juniper@cascrmdinet50rd-‐f1> show security nat static rule all node0: -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
SRX Troubleshooting Library Page 37
Total static-‐nat rules: 0 …
show security policies (from-‐zone|policy-‐name|to-‐zone) This command shows security policy configuration, sequence number, and status
ben@olive100> show security policies policy-‐name 1 From zone: blah, To zone: boo Policy: 1, State: enabled, Index: 4, Sequence number: 1 Source addresses: any Destination addresses: any Applications: any Action: permit
SRX Troubleshooting Library Page 38
Contacting JTAC To Open A Technical Support Case When contacting JTAC to open a technical case they will most likely require you to upload several diagnostic files.
Case Opening Procedure Creating a case via the web:
1. Visit http://www.juniper.net/cm 2. Log in with your Juniper Support Account (probably your e-‐mail address) 3. Click on "Create a Case" under the "My Cases and RMAs" section 4. Select the Juniper Platform with the issue 5. Fill in the serial number (use ‘show chassis hardware’) 6. Make sure "Technical Support Case" is checked 7. Click "Next" 8. Fill out a brief synopsis of the problem "My IDP quit working" 9. Select a priority (if it's critical I would create a bridge for me and JTAC to work with you on) 10. Fill out any additional detail on the problem in the Problem Description Form 11. Select the Platform experience the problem (if it's an IDP and NSM problem, select whichever you
believe it is, we can transfer it later) 12. Select the Code Release 13. Select the Version of Code your'e running 14. If you have a Remedy or Internal tracking number fill that in 15. Give the system name "nyorbgdpciyl-‐f1" 16. Verify the Serial Number is Correct 17. Add anyone that needs to be copied to the Additional Recipients 18. Select the follow up method as "Email Full Text Update” 19. Click "Create Case" 20. Upload the following files to the case
a. request support information command output b. messages log c. chassisd log d. jsrpd log e. Any other relevant packet-‐captures, logs, etc
21. If this is an emergency follow up with a call to JTAC @ 1-‐888-‐314-‐5822
request support information | save rsi_[date].txt ben@olive100> request support information | save rsi.txt Wrote 1567 lines of output to 'rsi.txt'
SRX Troubleshooting Library Page 39
Action Commands These commands can be used during the troubleshooting process, but be careful when you “request” anything from JUNOS, it typically involves downtime of some sort.
set chassis cluster cluster-‐id 1 node 1 reboot This command enables clustering on the SRX. After a reboot, the SRX will come up as node 1 in cluster-‐id 1. This command cannot be used twice. So if clustering is disabled, the first cluster-‐id can never be used again.
request chassis cluster failover redundancy-‐group 1 node 1 This command initiates a chassis cluster failover. The result of the failover will be make node 1 the primary node for redundancy-‐group 1.
request chassis cluster failover reset redundancy-‐group 1 This command resets the “manual reset” bit set when a manual failover is performed.
request system reboot This command reboots the current node of the SRX cluster.
request system halt (request system power-‐off) This command turns off the current node of the SRX cluster. This is needed when adding or removing SPCs from the SRX.
request routing-‐engine login node 1 This command logs into the other node from the current node of the SRX cluster.
request chassis pic fpc-‐slot 0 pic-‐slot 0 offline This command turns off a particular PIC. This is useful when replacing network cards.
request system software add (location of image) no-‐validate no-‐copy reboot This command loads a new version of JUNOS and reboots the SRX. This physical SRX will then boot with the new version. Be careful, SRX clusters need to be on the same version and can act very strangely if they are not.