5-1 © 2000, cisco systems, inc. chapter 5 applying cisco troubleshooting tools

5-1© 2000, Cisco Systems, Inc.

Chapter 5Chapter 5

Applying Cisco Troubleshooting Tools

Applying Cisco Troubleshooting Tools

© 2000, Cisco Systems, Inc. CIT 4.1—5-2

ObjectivesObjectives

Upon completion of this chapter, you will be able to:• Handle troubleshooting tools and minimize their impact

on a Cisco router’s switching type and data flow

• Identify and use the innate Cisco IOS software commands and debug utilities to filter, capture, and display protocol traffic flows

• (optional) Obtain protocol troubleshooting information by capturing and interpreting data with a third-party protocol analyzer


Apply Cisco IOS Software Troubleshooting Tools

Apply Cisco IOS Software Troubleshooting Tools

• show commands—snapshot problems with interfaces, media, or network performance

• debug commands—check the flow of protocol traffic for problems, protocol bugs, or misconfigurations

• ping commands—test the reachability of a protocol on a targeted remote router or other device

• trace commands—determine a step-by-step routing path to remote routers or other devices


Understanding the Output of Diagnostic Commands

Understanding the Output of Diagnostic Commands

Understand basic router architecture and processes

Switching processes

Routing processes

show buffersshow controllers

trace

debug

show interfaces ping

SiSi

TokenRing ATM

EthernetSerial FDDI


Why Accept the Impact of Cisco Troubleshooting Tools?

Why Accept the Impact of Cisco Troubleshooting Tools?

• You need tools that diagnose the targeted protocol behavior to troubleshoot network problems

• Understanding troubleshooting output requires an understanding of router architecture and processes

• Some diagnostic or control resources come at the expense of processing and switching efficiency

• Selective and temporary use of tools can minimize the impact of these tools on router performance

• To balance troubleshooting needs and router efficiency, you must operate the tools correctly


Routing and Switching Processes of Cisco Routers

Routing and Switching Processes of Cisco Routers

Routing functions identify the best path to a destination over an interface

Switching types function to move the packets, frames, or cells themselves

Understand the two different processes:


Routing ProcessRouting Process

101

102

103

106

105104

A

BATM

FDDI

TokenRing

C

D


Routing Process (cont.)Routing Process (cont.)

101

102

103

106

105104

A

BATM

FDDI

TokenRing

C

D

Update

Update

Update Update

Update


Routing Process (cont.)Routing Process (cont.)

101

102

103

106

105104

A

BATM

FDDI

TokenRing

C

D

Update

Update

Update Update

Update

Destination Network101102103104105106

Routing Table of Router B

Reachable InterfaceFDDI 0/0FDDI 0/0

Ethernet 1/0ATM 2/0ATM 2/0ATM 2/0


Switching ProcessSwitching Process

FDDI

102

BATM

103

104

PacketHEADER

Data

ETHERNET


FDDI

102

Switching Process (cont.)Switching Process (cont.)

BATM

103

104

PacketETHERNETHEADER





Data


FDDI

102

Switching Process (cont.)Switching Process (cont.)

BATM

103

104

PacketETHERNETHEADER





Data

FDDIHEADER

Packet


Switch Initialization—Generic Routers

Switch Initialization—Generic Routers

Route Processor

AutonomousSwitchCache

System Bus

Silicon SwitchProcessor

Fast SwitchCache

SSE

Silicon SwitchCache

Process Switching (Initialization)

Silicon Switching Autonomous Switching Fast Switching

Switching Types


Switching Initialization—Small to Mid-Range Routers

Switching Initialization—Small to Mid-Range Routers

NetworkInterface

200 Mbps

NetworkInterface

NetworkInterface

TokenRing

CPUMemory

FastSwitchingCache


Fast Switching

Switching Types

SharedMemory

High-SpeedAccelerator

CPU


Switching Initialization—Higher Series Routers

Switching Initialization—Higher Series Routers

Route/SwitchProcessor

(RSP)

CyBus

InterfaceProcessor

VersatileInterface

Processor

Fast SwitchCache

OptimumSwitch Cache

InterfaceProcessor


Fast Switching Optimum Switching

Switching Types


Example: Tracing Packet Flow on a Router

Example: Tracing Packet Flow on a Router

Silicon SwitchProcessor System Bus

Route Processor

Cisco Extended Bus(CxBus)

ATMFDDI

SSE



Route Processor


ATMFDDI

SSE

DataHDR A

Example: Tracing Packet Flow on a Router (cont.)




Route Processor


ATMFDDI

SSE

DataHDR A

DataHDR A





Route Processor


ATMFDDI

SSE

DataHDR A

DataHDR A

Destination ReachableInterface

MACHeader

Silicon Switching Cache





Route Processor


ATMFDDI

SSE

DataHDR A

DataHDR A


MACHeader

Silicon Switching Cache Autonomous Switching Cache


MACHeader




Tracing Packet Flow on a Router (cont.)


Silicon SwitchProcessor System

Bus


ATMFDDI

DataHDR A

Route Processor





Bus


ATMFDDI

DataHDR A

Route Processor

Layer 3HDR

Layer 3HDR





Bus


ATMFDDI

DataHDR ALayer 3

HDRLayer 3

HDR


MACHeader

Fast SwitchingCache





Bus


ATMFDDI

DataHDR A


MACHeader

Fast SwitchingCache

DataLayer 3HDR




System Bus


ATMFDDI

DataHDR A




System Bus


ATMFDDI

DataHDR A

DataHDR A




System Bus


ATMFDDI

DataHDR A

DataHDR A

DataHDR A


The Route Processor Handles Broadcasts and Special Processes

The Route Processor Handles Broadcasts and Special Processes

FFFF.FFFF.FFFF

• Debug filtering• Error logging to syslog• SNMP

Route Processor

Keepalives

• Protocol translations• Tunnel maintenence• Special queuing

Special processes like diagnostic or control functions come at the expense of processing and switching efficiency


Handling the Cisco IOS debug Troubleshooting Tool

Handling the Cisco IOS debug Troubleshooting Tool

• Be aware of its process switching implications

• Use to isolate problems, not to monitor network

• Focus scope of debug command with arguments

• Limit impact of checking with access lists

• Undo the debug after troubleshooting


debug Commands Use Process Switching

debug Commands Use Process Switching

Switching Types

Fast Switching

Optimum Switching

Net Flow Switching

4. debug commands use process switching

3. Problem diagnosis requires troubleshooting tools

2. Faster switching for subsequent packets

1. Initialization with process switching


Using the Cisco IOS debug Command

Using the Cisco IOS debug Command

Suggestions from debug users:

• The command service timestamps adds time stamps to debug output

• Do not use debug all; focus debug with specific arguments

• Use terminal monitor to see debug output from a Telnet terminal

• Consider debugging events, rather than debugging packets

• When you can, apply an access control list to filter debug


Using an Access List with debugUsing an Access List with debug

• Constrain focus of debug command in IP

• Specify source and destination option

• Specify an optional IP protocol or protocol port number

• Remove access list and undebug when finished

access-list 101 extendedpermit tcp any host 172.16.4.20 eq telnet establishedinterface ether 1ip access-group 101end

debug ip packet detail telnet 172.16.4.20


Undo debug after Troubleshooting Undo debug after Troubleshooting

• Use the no debug [argument} command to turn off a specific debug

• Use undebug all or no debug all command to turn off all debugging

• Turn off all debugging when you are finished troubleshooting

• Configure the router to restore any faster switching type

router# debug broadcast

Ethernet0: Broadcast ARPA, src 0000.0c00.6fa4, dst ffff.ffff.ffff, type 0x0800,data 4500002800000000FF11EA7B, len 60Serial3: Broadcast HDLC, size 64, type 0x800, flags 0x8F00Serial2: Broadcast PPP, size 128Serial7: Broadcast FRAME-RELAY, size 174, type 0x800, DLCI 7a

router# undebug allAll debugging has been turned offrouter#


Error Message LoggingError Message Logging

terminal monitor

Syslog Server

Console

Console


Error Message Logging (cont.)Error Message Logging (cont.)

terminal monitor

Syslog Server

Console

Console

logging on(default)



logging console

terminal monitor

Syslog Server

Console

Console

logging on(default)



logging console

terminal monitor

Syslog Server

Console

Console

logging on(default)

logging buffered(default)



logging console

terminal monitor

Syslog Server

Console

Console

logging on(default)


logging monitor



logging console

terminal monitor

Syslog Server

Console

Console

logging on(default)


logging monitor

logging ip-address



logging console

terminal monitor

Syslog Server

Console

Console

logging on(default)


logging monitor

logging ip-addresslogging trap


Limiting the Display of Error Messages

Limiting the Display of Error Messages

emergencies 0 System unusable LOG_EMERG

alerts 1 Immediate action needed LOG_ALERT

critical 2 Critical conditions LOG_CRIT

errors 3 Error conditions LOG_ERR

warnings 4 Warning conditions LOG_WARNING

notifications 5 Normal but significant condition LOG_NOTICE

informational 6 Informational messages only LOG_INFO

debugging 7 Debugging messages LOG_DEBUG

Level Name Level Description Syslog Definition

• Use logging console level command for console terminal

• Use logging monitor level command for other terminals

• Use logging trap level command for syslog server


show logging Commandshow logging Command

Router#show loggingSyslog logging: enabled Console logging: disabled Monitor logging: level debugging, 18 messages logged. Trap logging: level informational, 18 messages logged. Logging to 192.31.7.19SNMP logging: enabled, retransmission after 30 seconds 741 messages logged Logging to 131.108.1.27, 0/10 Logging to 131.108.1.111, 0/10 Logging to 131.108.2.63, 0/10


Reachability and Step-by-Step Path Tests

Reachability and Step-by-Step Path Tests

Test reachability:• ping ip• ping ipx

Test step-by-step path:• trace

These tests operate in two levels: user mode and privileged mode


ping Command (User) IPping Command (User) IP

Router> ping fredType escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.31.7.27, timeout is 2 seconds:!!!!!Success rate is 100 percent, round-trip min/avg/max = 1/3/4 ms

Router> ping 192.45.3.1Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.45.3.1, timeout is 2 seconds:.U.U.Success rate is 0 percent (0/5)


ping Command (Privileged) IPping Command (Privileged) IP

Router# pingProtocol [ip]:Target IP address: fredRepeat count [5]:Datagram size [100]:Timeout in seconds [2]Extended commands [n]: ySource address:Type of service [0]:Set DF bit in IP header? [no]:Data pattern [0xABCD]:Loose, Strict, Record, Timestamp, Verbose[none]: rNumber of hops [9]:Loose, Strict, Record, Timestamp, Verbose[RV]:Sweep range of sizes [n]:Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 131.108.1.115, timeout is 2 seconds:Packet has IP options: Total option bytes=39, padded length=40Record route: <*> 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0

0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0

- More -


ping Command (Privileged) IP (cont.)

ping Command (Privileged) IP (cont.)

- cont-

The following display is a detail of the Echo packet section:

0 in 4 ms. Received packet has optionsTotal option bytes= 40, padded length=40Record route: 160.89.80.31 131.108.6.10 131.108.1.7 131.108.1.115

131.108.1.115 131.108.6.7 160.89.80.240 160.89.80.31 <*> 0.0.0.0End of list


131.108.1.115 131.108.6.7 160.89.80.240 160.89.80.31 <*> 0.0.0.0End of list

::


131.108.1.115 131.108.6.7 160.89.80.240 160.89.80.31 <*> 0.0.0.0End of list

Success rate is 100 percent, round-trip min/avg/max = 4/5/8 ms


ping Command (User) IPXping Command (User) IPX

Router> ping ipx 211.0000.0c01.f4cfType escape sequence to abort.Sending 5, 100-byte Novell Echoes to 211.0000.0c01.f4cf, timeout is 2 seconds:•••••Success rate is 0 percent (0/5)


ping Command (Privileged) IPXping Command (Privileged) IPX

Router# pingProtocol [ip]: ipxTarget Novell Address: 211.0000.0c01.f4cfRepeat count [5]:Datagram Size [100]:Timeout in seconds [2]:Verbose [n]:Novell Standard Echo [n]:Type escape sequence to abort.Sending 5 100-byte Novell echoes to 211.0000.0c01.f4cf, timeout is 2 seconds.!!!!!Success rate is 100%, round trip min/avg/max = 1/2/4 ms.


trace Command (User) IPtrace Command (User) IP

Router# trace ip ABA.NYC.milType escape sequence to abort.Tracing the route to ABA.NYC.mil (26.0.0.73)1 DEBRIS.CISCO.COM (131.108.1.6) 1000 msec 8 msec 4 msec2 BARRNET-GW.CISCO.COM (131.108.16.2) 8 msec 8 msec 8 msec3 EXTERNAL-A-GATEWAY.STANFORD.EDU (192.42.110.225) 8 msec 4 msec 4 msec4 BB2.SU.BARRNET.NET (131.119.254.6) 8 msec 8 msec 8 msec5 SU.ARC.BARRNET.NET (131.119.3.8) 12 msec 12 msec 8 msec6 MOFFETT-FLD-MB.in.MIL (192.52.195.1) 216 msec 120 msec 132 msec7 ABA.NYC.mil (26.0.0.73) 412 msec 628 msec 664 msec


trace Command (Privileged) IPtrace Command (Privileged) IP

Router# traceProtocol [ip]:Target IP address: mit.eduSource address:Numeric display [n]:Timeout in seconds [3]:Probe count [3]:Minimum Time to Live [1]:Maximum Time to Live [30]:Port number[33434]:Loose, Strict, Record, Timestamp, Verbose[none]:Type escape sequence to abort.Tracing the route to MIT.EDU (18.72.2.1)1 ICM-DC-2-V1.ICP.NET (192.108.209.17) 72 msec 72 msec 88 msec2 ICM-FIX-E-H0-T3.ICP.NET (192.157.65.122) 80 msec 128 msec 80 msec3 192.203.229.246 540 msec 88 msec 84 msec4 T3-2.WASHINGTON-DC-CNSS58.T3.ANS.NET (140.222.58.3) 84 msec 116 msec 88 msec5 T3-3.WASHINGTON-DC-CNSS56.T3.ANS.NET (140.222.56.4) 80 msec 132 msec 88 msec6 T3-0.NEW-YORK-CNSS32.T3.ANS.NET (140.222.32.1) 92 msec 132 msec 88 msec7 T3-0.HARTFORD-CNSS48.T3.ANS.NET (140.222.48.1) 88 msec 88 msec 88 msec8 T3-0.ENSS134.T3.ANS.NET (140.222.134.1) 92 msec 128 msec 92 msec9 W91-CISCO-EXTERNAL-FDDI.MIT.EDU (192.233.33.1) 92 msec 92 msec 112 msec10 E40-RTR-FDDI.MIT.EDU (18.168.0.2) 92 msec 120 msec 96 msec11 MIT.EDU (18.72.2.1) 96 msec 92 msec 96 msec


Information Needed by Technical Support

Information Needed by Technical Support

1. General informationProblem historyProblem symptomsshow version

5. Optional informationDebug capturesProtocol analyzer

2. Crash or hang problemsshow stackcore dump

4. Loss of functionalityproblems

show interfacesshow protocolshow protocol traffic

3. Lost data or performanceproblems

show interfacesshow buffershow memoryshow processshow protocolshow protocol traffic


show version Commandshow version Command

Router>show versionCisco Internetwork Operating System SoftwareIOS (tm) GS Software (GS7-K-M), Version 11.0(9), RELEASE SOFTWARE (fc1)Copyright (c) 1986-1996 by cisco Systems, Inc.Compiled Tue 11-Jun-96 03:52 by tstevensImage text-base: 0x00001000, data-base: 0x007614F0

ROM: System Bootstrap, Version 5.2(2), RELEASE SOFTWAREROM: GS Software (GS7), Version 10.2(2), RELEASE SOFTWARE (fc1)

Router uptime is 2 days, 20 hours, 26 minutesSystem restarted by error - Software forced crash, PC 0x1CF82C at 20:25:38 PDT Mon Aug 5 1996System image file is "images/gs7-k-mz.110-9", booted via flash

cisco RP1 (68040) processor (revision B0) with 16384K bytes of memory.Processor board ID 00130334G.703/E1 software, Version 1.0.Bridging software.X.25 software, Version 2.0, NET2, BFE and GOSIP compliant.Primary Rate ISDN software, Version 1.0.1 Silicon Switch Processor.1 EIP controller (6 Ethernet).1 FEIP controller (2 FastEthernet).6 Ethernet/IEEE 802.3 interfaces.2 FastEthernet/IEEE 802.3 interfaces.48 Serial network interfaces.128 bytes of non-volatile configuration memory.4096K bytes of flash memory sized on embedded flash.

Configuration register is 0x102 Router>


Buffers and Queues—Cisco 7xx0 Routers

Buffers and Queues—Cisco 7xx0 Routers

InterfaceBuffers

SystemBuffers

InputQueue

OutputQueue

show controllers cxbus

Silicon Switch

Processor

Route ProcessorDrops

Misses, failures (no memory)

show buffers

System Bus

CxBus

HardwareBuffers

Pointers toInterface Queues

Overruns Underruns

IgnoredR Q x

T Q x


Buffers and Queues—4000/3000/2500

Buffers and Queues—4000/3000/2500

NetworkInterface

200 Mbps

HardwareBuffers

Overruns

TokenRing

CPUMemory

Drops

Underruns

Ignored

High-SpeedAccelerator

CPU

InterfaceBuffers

SystemBuffers

SharedMemory


show buffers Commandshow buffers Command

Router# show buffers Buffer elements: 398 in free list (500 max allowed) 1266 hits, 0 misses, 0 created Public buffer pools:Small buffers, 104 bytes (total 50, permanent 50): 50 in free list (20 min, 150 max allowed) 551 hits, 0 misses, 0 trims, 0 createdMiddle buffers, 600 bytes (total 25, permanent 25): 25 in free list (10 min, 150 max allowed) 39 hits, 0 misses, 0 trims, 0 createdBig buffers, 1524 bytes (total 50, permanent 50): 49 in free list (5 min, 150 max allowed) 27 hits, 0 misses, 0 trims, 0 createdVeryBig buffers, 4520 bytes (total 10, permanent 10): 10 in free list (0 min, 100 max allowed) 0 hits, 0 misses, 0 trims, 0 createdLarge buffers, 5024 bytes (total 0, permanent 0): 0 in free list (0 min, 10 max allowed) 0 hits, 0 misses, 0 trims, 0 createdHuge buffers, 18024 bytes (total 0, permanent 0): 0 in free list (0 min, 4 max allowed) 0 hits, 0 misses, 0 trims, 0 created Interface buffer pools:Ethernet0 buffers, 1524 bytes (total 64, permanent 64):


show memory Commandshow memory Command

Router#show memoryHead Free Start Total Bytes Used Bytes Free BytesProcessor AA0A8 E42D8 3497816 308700 3189116Multibus 2000000 2000000 32768 0 32768

Address Bytes Prev. Next Free? PrevF NextF Alloc PC WhatAA0A8 916 0 AA43C 7ACE *Init*AA43C 2024 AA0A8 AAC24 AD2E *Init*AAC24 536 AA43C AAE3C AD58 *Init*AAE3C 2024 AAC24 AB624 49BC *Init*AB624 72 AAE3C AB66C 248E0 *Init*AB66C 44 AB624 AB698 3614C *Init*AB698 152 AB66C AB730 1CFC *Init*AB730 2024 AB698 ABF18 1D20 *Init*ABF18 152 AB730 ABFB0 1CFC *Init*ABFB0 2024 ABF18 AC798 1D20 *Init*AC798 100 ABFB0 AC7FC 3F2FE LoggerAC7FC 152 AC798 AC894 y E2568 D74E8 74E12 TCP ProtocolsAC894 44 AC7FC AC8C0 4BCC *Sched*AC8C0 1880 AC894 AD018 y D74E8 D7134 74E3C TCP ProtocolsAD018 104 AC8C0 AD080 5126 *Init*AD080 2024 AD018 AD868 67E6 *Init*AD868 348 AD080 AD9C4 54BA *Init*AD9C4 348 AD868 ADB20 54BA *Init*ADB20 348 AD9C4 ADC7C 54BA *Init*ADC7C 348 ADB20 ADDD8 54BA *Init*


show processes commandshow processes command

CPU utilization for one minute: 38%; for five minutes: 37% PID Q T PC Runtime (ms) Invoked uSecs Stacks TTY Process 1 M E 122DE 62812 4897 12826 780/1000 0 Net Background 2 M E 22842 8 19 421 804/1000 0 Logger 809 M E 74AF0 272808 489888 556 1504/2000 36 Exec 4 H E 67C0 373540 630248 592 628/900 0 IP Input 5 M E 3E124 26044 630201 41 824/1000 0 IP Protocols 6 M E 46BA2 592 255178 2 794/1000 0 TCP Timer 7 L E 47CE6 1736 1635 1061 776/1000 0 TCP Protocols 8 L E 67C0 0 1 0 958/1000 0 ARP Input 813 M * 768 384 93 4129 1456/2000 42 Virtual Exec 10 M E 3F51E 0 1 0 894/1000 0 BOOTP Server 11 H E 67C0 25096 194823 128 426/500 0 Net Input 12 M T 36FA 5420 277303 19 850/1000 0 TTY Background 13 L E 5444E 65996 24907 2649 686/1000 0 SNMP Server 14 M E 6E842 0 1 0 966/1000 0 Serial Line IP

Router#show processes

Add argument cpu to show detailed CPU utilization

Add argument memory to show memory usage


show controllers cxbus Command

show controllers cxbus Command

Router# show controllers cxbus

Switch Processor 5, hardware version 11.1, microcode version 172.6Microcode loaded from system512 Kbytes of main memory, 128 Kbytes cache memory75 1520 byte buffers, 86 4484 byte buffersRestarts: 0 line down, 0 hung output, 0 controller error

CIP 3, hardware version 1.1, microcode version 170.1Microcode loaded from systemCPU utilization 7%, sram 145600/512K, dram 86688/2MInterface 24 - Channel 3/043 buffer RX queue threshold, 61 buffer TX queue limit, buffer size 4484ift 0007, rql 32, tq 0000 0468, tql 61Transmitter delay is 0 microseconds

Interface 25 - Channel 3/143 buffer RX queue threshold, 61 buffer TX queue limit, buffer size 4484ift 0007, rql 34, tq 0000 0000, tql 61Transmitter delay is 0 microseconds


show stacks Commandshow stacks Command

Router# show stacksMinimum process stacks:Free/Size Name652/1000 Router Init726/1000 Init744/1000 BGP Open686/1200 Virtual Exec

Interrupt level stacks:Level Called Free/Size Name1 0 1000/1000 env-flash3 738 900/1000 Multiport Communications Interfaces5 178 970/1000 Console UARTSystem was restarted by bus error at PC 0xAD1F4, address 0xD0D0D1AGS Software (GS3), Version 10.2Compiled Tue 11-Aug-94 13:27 by jthomasStack trace from system failure:FP: 0x29C158, RA: 0xACFD4FP: 0x29C184, RA: 0xAD20CFP: 0x29C1B0, RA: 0xACFD4FP: 0x29C1DC, RA: 0xAD304FP: 0x29C1F8, RA: 0xAF774FP: 0x29C214, RA: 0xAF83EFP: 0x29C228, RA: 0x3E0CAFP: 0x29C244, RA: 0x3BD3C


Core DumpsCore Dumps

TFTP Server

Router

System Memory

write coreexception dump ip-address

Use write core to execute dump core immediately

Use exeption dump to set up for some future crash


SummarySummary

After completing this chapter, you should be able to do the following:• Apply the innate Cisco IOS troubleshooting tools:

– show, debug, ping, and trace commands

• Be aware of the impact of using extended ping and debug on the system

• For debug, focus the command with arguments and, if you can, with access lists

• When you are done troubleshooting with debug, turn it off with no debug or undebug command

• Use other tools (for example, protocol analyzers) if you need them

5-1 © 2000, cisco systems, inc. chapter 5 applying cisco troubleshooting tools

Documents