1-tshooting models, methods, tools and tips

7/29/2019 1-Tshooting Models, Methods, Tools and Tips

1/30

The Bryant Advantage CCNP TShoot Study Guide

Chris Bryant, CCIE #12933 http://www.thebryantadvantage.com/


2/30

When someone walks up to you and tells you their printer is broken, orthey call you and say they can't hit the Net, you don't usually go off for awhile and consider which t-shooting approach you're going to use.

You just go fix it.

And in doing so, you're actually using one of the following models...

Spot The Difference: I like to use this one with suspected passwordmismatches. Just copy the suspicious part of each config and put themright next to each other in a single Notepad. Works like a charm and it'smuch easier to spot a problem this way than trying to scroll up and downin different configs. (Notepad is not available in the exam room,naturally.)

Divide And Conquer: You can use this in one of two ways. If you'relucky enough to have several admins available to work on a singleproblem (stop laughing!), you can divide the t-shooting labor up and findthe issue quickly.

Now, heaven forbid you have to work on a problem by yourself-- but ifyou do, you can start using tools such as ping and traceroute to quicklyeliminate some potential issues.

Top-Down and Bottom-Up both refer to using the OSI model to t-shoot.The bottom-up method is my personal favorite; I used it many a time ontech support calls and many of you likely do the same. Why waste timeon more esoteric solutions if you don't know whether the power's on ornot?

I always begin t-shooting with Layer 1 - the Physical layer - when theissue involves a host device such as a workstation or printer. If thepower's not working, nothing's working.

Follow The Path is a good approach for connectivity issues, particularlyover a WAN. In a simple host-server path, we have several potentialproblem spots...

The host itself (good spot for a "self-ping" to 127.0.0.1)

The default gateway

Any switches and routers along the way

The server itself

I like to start with a self-ping and go from there. If the local host can't pingitself, you know there's a local TCP/IP issue. If it can, and it can't ping thedefault gateway, you know to check the local network settings. If it can,

just keep pinging until you get a negative result, and then you know where

to start troubleshooting.


3/30

Swap It Outis just what it sounds like - swapping hardware to see if that'sthe issue. Even though this is Layer 1 stuff, it's not the first step I like totake.

Here's my rather informal but highly effective t-shooting path:

What exactly isn't working? (Love trouble tickets that say "thisdoesn't work" or something equally helpful. What is "this"? What

does "doesn't work" mean?)

Get the info you need to fix it. On Cisco devices, that's where yourshowand debugcommands come in. Combine your debugs withshow interface, show process cpu,pings (extended and regular) andtraceroute, and you can get any info you need to resolve anyproblem.

If you have multiple potential points of trouble (especially in a WAN),localize the issue. Where's the problem?

Fix it.

Document the problem and the fix. ( You don't have to like this step;you just have to do it.)

Or put another way...

Hear about the problem

Figure out what the problem is

Fix the problem

Simple!

And speaking of documentation ....

Creating A Network Baseline And Map

Documenting certain network statistics before a problem occurs can be a

big help in t-shooting your network.

For example, let's say you suspect that a router CPU is working overtime.You check the load and it's running at 55%. It's impossible to know if thisis unusual or usual without documenting that and other network stats on aregular basis - in other words, creating a network baseline.

In addition to this baseline, a thorough and accurate network map is amuch in today's world. I placed emphasis on those terms since I seemany network maps that are neither. At the least, the maps should

include..


4/30

The physical andlogical network topologies, including protocolgroups (OSPF areas, EIGRP autonomous systems, HSRP groups,etc.)

The cabling connections should be clearly marked, along with legibleIP addresses and masks

Location of hardware along with an inventory

A New Look At Old Friends

You've used several of the following commands for quite a while at thispoint in your career, but there are some important options you might nothave seen or used. Let's take a detailed look at .....

Ping

We all know the basics of this command - ping the destination with fiveICMP Echo, and hopefully we get five ICMP Echo Replies!

R1#ping 172.23.23.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.23.23.2, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/72 ms

However, when we don'tget those replies, and get something like thisinstead ...

R1#ping 20.1.1.1



.....

Success rate is 0 percent (0/5)

.. the basic ping result really doesn't tell us much of anything except "Hey,this failed." No hint as to where the issue is, or what the issue is.

That's where extended ping comes in.

For extended ping, you have to be in enable mode, and just enter theword ping followed by the enter key.

R1#ping

Protocol [ip]:

You'll then be presented with a long list of options that may remind you ofsetup mode. "Do you want to do this? How about this? How aboutthis?" It's much shorter than setup mode, so hang in there.

To accept the default shown in the brackets, just hit the enter key. Here's

the full list when you enter "no" for the extended commands and sweep


5/30

range of sizes. Just for fun, I'll enter some non-default values.

R1#ping

Protocol [ip]:

Target IP address: 172.23.23.2

Repeat count [5]: 1000

Datagram size [100]: 10

% A decimal number between 36 and 18024.

Datagram size [100]:

Timeout in seconds [2]: 1

Extended commands [n]:

Sweep range of sizes [n]:Type escape sequence to abort.


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.


All self-explanatory -- but did you notice that the ping stopped after 40packets, even though I entered 1000 for the number of pings?

That's because I entered the super-secret keystroke to terminate bothpings and traceroutes. Here's the IOS Help "help" with that:


That's great, as long as you know what that sequence is.

Don't tell anyone I told you, but it's twice, one right afterthe other.

I'll run the extended ping again, this time saying "yes" to the extendedcommands so we can see what they are.

Extended commands [n]: ySource address or interface:

Type of service [0]:

Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

The one you're most likely to use here is the source address or interface.By default, pings will leave via the interface that other packets would useto reach the ping destination - which certainly makes perfect sense.

We can change that default to test connectivity from a network that ispresent on the router, but not on the interface the pings will use to exit therouter (the "egress interface").

Both routers are running RIP and are communicating over the172.12.123.0 /24 network. All labs in the course use the router numberas the last octet. We'll have R1 ping R3 with both a regular and extendedping, and all is well.

R1#ping 172.12.123.3



6/30


!!!!!


R1#ping

Protocol [ip]:


Repeat count [5]: 100Datagram size [100]:

Timeout in seconds [2]: 1

Extended commands [n]: ySource address or interface: 172.12.123.1Type of service [0]:Set DF bit in IP header? [no]:

Validate reply data? [no]:Data pattern [0xABCD]:Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:Type escape sequence to abort.


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.


I used to terminate the ping before all 100 ICMPpackets were sent. It's normal to see a ping failure at the very end of a

terminated ping (65/66).

Let's add a new segment is added to R1.

You've gotten a call that no one on the new segment can communicatewith R3. However, if you try to ping R3 from R1, it goes through like it didbefore. but if you send a regular ping from R1 to R3 right now, it still goesthrough. debug ip icmp shows why....

R1#ping 172.12.123.3



!!!!!


R1#

23:01:33: ICMP: echo reply rcvd, src 172.12.123.3, dst 172.12.123.1


7/30

23:01:33: ICMP: echo reply rcvd, src 172.12.123.3, dst 172.12.123.123:01:33: ICMP: echo reply rcvd, src 172.12.123.3, dst 172.12.123.123:01:33: ICMP: echo reply rcvd, src 172.12.123.3, dst 172.12.123.123:01:33: ICMP: echo reply rcvd, src 172.12.123.3, dst 172.12.123.1

By default, the ping will be sourced from the exit interface on the localrouter. Makes sense, but it can lead to confusing ping results.

Another good t-shoot step here would be to start from R3 and see if itcould ping the next-closest interface - - that is, R1's Serial interface.

The way to send a ping from R1's ethernetinterface to R3 is to useextended ping with extended commands.

R1#ping

Protocol [ip]:


Repeat count [5]:

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]: y

Source address or interface: 10.1.1.1

Type of service [0]:Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:



.....


Extended ping has shown us that R1's Serial0 interface can ping R3, but

R1's Ethernet0 cannot. Don't make the assumption that you have 100%IP connectivity on a router just because one interface can ping a givendestination.

Here, the problem is that R1's Ethernet0 interface wasn't a part of the RIPdeployment.

R1#show ip protocols

Routing Protocol is "rip"

Sending updates every 30 seconds, next due in 11 seconds

Invalid after 180 seconds, hold down 180, flushed after 240

Outgoing update filter list for all interfaces is not set

Incoming update filter list for all interfaces is not setRedistributing: rip

Default version control: send version 2, receive version 2

Interface Send Recv Triggered RIP Key-chain

Serial0 2 2

Serial1 2 2

Automatic network summarization is not in effect

Maximum path: 4

Routing for Networks:

172.12.0.0

Routing Information Sources:

Gateway Distance Last Update

172.12.123.3 120 00:00:12

172.12.123.2 120 00:50:07

Distance: (default is 120)


8/30

Adding that network allows that subnet to communicate with R3.

R1(config)#router ripR1(config-router)#network 10.0.0.0R1(config-router)#^ZR1#pin23:07:10: %SYS-5-CONFIG_I: Configured from console by consoleR1#pingProtocol [ip]:Target IP address: 172.12.123.3

Repeat count [5]:Datagram size [100]:

Timeout in seconds [2]:Extended commands [n]: ySource address or interface: 10.1.1.1Type of service [0]:Set DF bit in IP header? [no]:Validate reply data? [no]:Data pattern [0xABCD]:Loose, Strict, Record, Timestamp, Verbose[none]:Sweep range of sizes [n]:Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 72/72/72 msR1#23:07:24: ICMP: echo reply rcvd, src 172.12.123.3, dst 10.1.1.123:07:24: ICMP: echo reply rcvd, src 172.12.123.3, dst 10.1.1.123:07:24: ICMP: echo reply rcvd, src 172.12.123.3, dst 10.1.1.123:07:24: ICMP: echo reply rcvd, src 172.12.123.3, dst 10.1.1.123:07:24: ICMP: echo reply rcvd, src 172.12.123.3, dst 10.1.1.1

The pings are correctly sourced from 10.1.1.1, and the incoming echoreplies are indeed destined for that address.

Let's look at a few of the other extended ping options:

Protocol, Target IP address, repeat count, and datagram size are all self-explanatory and defaults are shown above in the router configurations.

The timeoutinterval specifies how quickly the echo reply packet must bereceived in order for the ping to be considered successful.

set DF bit in IP header? refers to the Don't Fragment bit and whether itshould be set on the ping packets. If you suspect an MTU issue and wantto find the lowest MTU value along the path, set this to "yes". That will

prevent a router from fragmenting the ping packet, and instead that routerwill return an error message.

This command, along with the sweep range of sizes option, are excellentways to find out about downstream MTU settings. It can also result inseeing the letter "M" in your ping results -- and here are some of the moreunusual and unwanted ping result characters.

U: Destination unreachable. You may see this one when the localrouter has an entry for a remote network, but one of the downstream

routers doesn't have that route.


9/30

Q: Source Quench. Love that name. Just means the receivingrouter's too darn busy to answer.

M: Can't Fragment. (can be a result of using the DF bit being set asshown above)

&: Packet Time To Live (TTL) has been exceeded

?: As you'd expect, this is an unknown packet type.

Of these, that "U" will show up most often - and sometimes in combinationwith another character. Let's walk through a lab that will show you what I

mean - and we'll tie an important debug in as well.

The networks used in this section:

Frame Relay (Serial interfaces, all routers): 172.12.123.x /24 R2's loopback interface: 2.2.2.2 /24 R3's loopback interface: 3.3.3.3 /24

R1 is the hub of this network and R2 and R3 are the spokes. Each spokecan ping the other spoke's Serial interface:

R3#ping 172.12.123.2

Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 172.12.123.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 132/136/148 ms

R2#ping 172.12.123.3

Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 132/132/132 ms

It's important to remember that in a hub-and-spoke network such as this,all spoke-to-spoke traffic goes through the hub - in this case, R1.


10/30

Right now, neither R2 nor R3 can ping the other's loopback interface.

R2#ping 3.3.3.3



.....


R3#ping 2.2.2.2



.....Success rate is 0 percent (0/5)

Let's run that last ping again while running debug ip packet.

R3#ping 2.2.2.2



23:40:39: IP: s=3.3.3.3 (local), d=2.2.2.2, len 100, unroutable.


23:40:43: IP: s=3.3.3.3 (local), d=2.2.2.2, len 100, unroutable.23:40:45: IP: s=3.3.3.3 (local), d=2.2.2.2, len 100, unroutable.


11/30



This result means the packets aren't even leaving the local router, and weknow why - with no dynamic routing protocol in place, there's no way forR3 to possibly know about R2's loopback interface, since there's no entryin R3's routing table for that network.

Let's put one there with the ip route command.

R3(config)#ip route 2.2.2.2 255.255.255.255 172.12.123.1

Now R3 has an entry for that network ...

R3#show ip route

< code table removed >

Gateway of last resort is not set

2.0.0.0/32 is subnetted, 1 subnets

S 2.2.2.2 [1/0] via 172.12.123.13.0.0.0/32 is subnetted, 1 subnets

C 3.3.3.3 is directly connected, Loopback0


C 172.12.123.0 is directly connected, Serial0

... but still can't ping R2's loopback. Note the strange ping result, acombination of U and periods.

R3#ping 2.2.2.2


Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:U.U.U


Those "U"s are a downstream router saying "I don't know how the $*&%to get to this network." debug ip packetshows the packets are leavingR3, and these replies aren't coming from R2 -- they're coming from R1(172.12.123.1).

R3#ping 2.2.2.2

Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:U23:43:42: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:43:42: IP: s=172.12.123.1 (Serial0), d=172.12.123.3 (Serial0), len 56, rcvd 323:43:42: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending.U23:43:44: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:43:44: IP: s=172.12.123.1 (Serial0), d=172.12.123.3 (Serial0), len 56, rcvd 323:43:44: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending.USuccess rate is 0 percent (0/5)R3#23:43:46: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:43:47: IP: s=172.12.123.1 (Serial0), d=172.12.123.3 (Serial0), len 56, rcvd 3

Even though R3 has an entry for R2's loopback network, R1 does not.


12/30

R1#show ip route




We'll need a static route on R1...

R1(config)#ip route 2.2.2.2 255.255.255.255 172.12.123.2

R1#show ip route



S 2.2.2.2 [1/0] via 172.12.123.2



... and then the pings sent from R3 are successful. Note the replies are

now from 2.2.2.2.

R3#debug ip packetIP packet debugging is onR3#ping 2.2.2.2

Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 140/141/144 msR3#23:46:24: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:46:25: IP: s=2.2.2.2 (Serial0), d=172.12.123.3 (Serial0), len 100, rcvd 323:46:25: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:46:25: IP: s=2.2.2.2 (Serial0), d=172.12.123.3 (Serial0), len 100, rcvd 323:46:25: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:46:25: IP: s=2.2.2.2 (Serial0), d=172.12.123.3 (Serial0), len 100, rcvd 323:46:25: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0), len 100, sending23:46:25: IP: s=2.2.2.2 (Serial0), d=172.12.123.3 (Serial0), len 100, rcvd 323:46:25: IP: s=172.12.123.3 (local), d=2.2.2.2 (Serial0)R3#, len 100, sending23:46:25: IP: s=2.2.2.2 (Serial0), d=172.12.123.3 (Serial0), len 100, rcvd 3

That's where the debug ip packet command reallycomes in handy. Weknow the first two ping attempts failed, but they failed for very different

reasons -- reasons we couldn't see with ping, but could with that debug.

Speaking ofdebug ip packet....

I've seen debug ip packetoverwhelm a router in a lab, so you can imaginewhat you can do to a router with this command in a production network.There's an option with this command you should know about....

R1#debug ip packet ?

Access list

Access list (expanded range)

detail Print more debugging detail


13/30

R1#debug ip packet

... and it's not the detailoption. That's a lotof detail.

Instead, consider using the ACL option. You can use either standard orextended ACLs with this command, and this will show you only thepackets that match the list you configured.

Not only is this easier on the router, it's a lot easier on your eyes.

Traceroute

This one's simple enough to use - just follow the command with the IPaddress of the remote device, and you'll see the IP addresses of therouters along the path. When you see asterisks start to appear, you'llknow you've arrived at the problem area.

BTW - use the same escape sequence here that you used for pings.

show processes cpu

Great command to t-shoot suspected CPU overloads. This commandgives you a lot of detail, but the info you need to decide whether tocontinue to pursue CPU issues is right at the top:

R1#show processes cpuCPU utilization for five seconds: 2%/0%; one minute: 2%; five minutes: 0%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process1 4 36 111 0.00% 0.00% 0.00% 0 Load Meter 2 1840 182 10109 2.78% 1.98% 0.55% 0 Exec3 884 58 15241 0.00% 0.21% 0.12% 0 Check heaps

If those numbers don't match up with your baseline, and seem high, youmay be on to something. If they do match up and/or seem low, you knowthis isn't the issue.

That's also helpful info, since t-shooting isn't just knowing what to fix, it'salso knowing what doesn't need fixing.

show interface

You've been looking at this one since The Dawn Of Man, but there's someinfo in the middle that you might be overlooking.

R1#show interface serial 0

Serial0 is up, line protocol is up

Hardware is HD64570

Internet address is 172.12.123.1/24

MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation FRAME-RELAY, loopback not set

Keepalive set (10 sec)

LMI enq sent 3, LMI stat recvd 4, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 1023 LMI type is CISCO frame relay DTE

Broadcast queue 0/64, broadcasts sent/dropped 0/0, interface broadcasts


14/30

0

Last input 00:00:05, output 00:00:05, output hang never

Last clearing of "show interface" counters 00:00:51

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: weighted fair

Output queue: 0/1000/64/0 (size/max total/threshold/drops)

Conversations 0/1/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

4 packets input, 84 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

6 packets output, 79 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up

We get plenty of helpful L1 and L2 feedback from this commands, with thePhysical layer info right at the top..

R1#show interface serial 0

Serial0 is up, line protocol is up

... followed closely by L2 information, in this case Frame Relay:

Encapsulation FRAME-RELAY, loopback not set

Keepalive set (10 sec)

LMI enq sent 3, LMI stat recvd 4, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 1023 LMI type is CISCO frame relay DTE

Realistically speaking, you and I tend to stop looking at the output hereonce the Frame Relay info ends.

Watch your input and output drops and errors, though, particularly if thepercentage of drops and errors increases over time.

Piping

You can also "pipe" commands to narrow the output down a bit - andperform some other helpful tasks at the same time.

include - Earlier, when we ran show processes cpu, we actually ended upwith 57 lines of output below the 5-second, 1-minute, and 5-minute

summaries. Let's say you only wanted the output for the lineARP Input.Here's how you use piping to make this happen:

R1#show processes cpu | include ARP Input

8 20 186 107 0.00% 0.00% 0.00% 0 ARP Input

34 0 15 0 0.00% 0.00% 0.00% 0 RARP Input

R1#

R1#

R1#show processes cpu | include ARP input

R1#

Case sensitive, eh? Sure is - when I spelled input with a lower-case "i",

we received no output.


15/30

If there's an include option, you know there has to be an exclude option.Let's say you're on a Cisco switch and want to run show ip interface brief,but want to see only the interfaces with a state of "down". To exclude theones that are up...

SW1#show ip int brief | exclude upInterface IP-Address OK? Method StatusocolVlan1 unassigned YES unset administratively down

FastEthernet0/5 unassigned YES unset down







begin -- One of my favorites. Want to look at only the VTY line config inyour running configuration? Filter show run like this:

R1#show run | begin line vty

line vty 0 4

password vonraschke

login

!

end

Don't panic if that output doesn't show up immediately - it can take a fewseconds, maybe more than a few if your running config is large.

redirect , appendand tee - all three commands direct the output of thecommand to a specified TFTP server, but with important differences.

redirectdoes not show the output of the command on the screen, itsimply performs the redirect

tee both displays the command output on the screen and sends that

same output to the specified TFTP server. A single exclamationpoint above the displayed output means the output was successfullysent to the TFTP server.

appenddoes just that - the output is appended to the specified file onthe TFTP server.

There's no right or wrong to how often you use these, if at all, after youpass the TShoot exam. Some admins never use them, some use themoften. That's up to you.

Let's head back to our models for just a few minutes.


16/30

Some Formal Troubleshooting Models

FCAPS

Courtesy of our friends at Wikipedia, here's the classic definition ofFCAPS:

"FCAPS is the ISO Telecommunications Management Network model and

framework for network management. FCAPS is an acronym for Fault,Configuration, Accounting, Performance, Security, the managementcategories into which the ISO model defines network tasks. In non-billingorganizations Accounting is sometimes replaced with Administration."

Source: http://en.wikipedia.org/wiki/FCAPS

Take just a few minutes to read that doc - here are some high points:

Fault management includes much of what you think it would - "recognize,isolate, correct, and log faults that occur in the network" - and it also

includes collecting information from the network devices to predict faults.

Configuration management isn't just about your configs - it also includesstoring configs and log results.

The rest of the management is pretty much what they sound like --

Accounting management is just what it sounds like -- usage istracked and billed to their department.

Performance management involves collecting and tracking data tomake sure the network's performing at its max (preferably withoutgoing past its maximum)

Security management - as a great philosopher once said, 'Nuff Said!

The Information Technology Infrastructure Librarywas once described bya friend as "a big pile of best practices", and I'll go along with that. Thefollowing links aren't required reading for the Tshoot exam, but they dogive you more of a feel for what the ITIL is all about:

http://en.wikipedia.org/wiki/Information_Technology_Infrastructure_Library

http://www.itil-officialsite.com/

The PPDIOO model (pronounced "poo-poo-pee-do") is the official namefor the Cisco network design process, consisting of these self-explanatorysteps ...

Prepare

Plan


17/30

Design

Implement

Operate

Optimize

BTW, it's not really pronounced "poo-poo-pee-do".

But it shouldbe.

Finally, the Telecommunications Management Network is an ITU-Tprotocol model that "provides a framework for achieving interconnectivityand communication across heterogeneous operations systems andtelecommunications networks".

http://en.wikipedia.org/wiki/Telecommunications_Management_Network

Let's take a look at some important network t-shooting features - and we'llstart with the answer to a musical question...

We Really Better Know What Time It Is

It's vitalfor our routers and switches to have a synchronized, central timesource. Having an accurate time source for our network allows thefollowing to work correctly:

Router syslogs and timestamps will have accurate and synched time

throughout the network, making our t-shooting much less frustrating

Digital certificates rely on accurate and synched time

If you're using any kind of accounting in your network, accurate andsynched time is a necessity. You haven't lived until you bill adepartment for 87 days' usage of a network resource - in one month.

The Network Time Protocol allows us to specify time sources for ourrouters. A Cisco router can be configured to get its time from anotherrouter in the same network, or an external time source.

At the very top of the NTP hierarchy, we have stratum-0 devices. Timeservers at this level are typically atomic clocks, and you cannot configurea Cisco router to get its time directly from a stratum-0 server.


18/30

Stratum-1 servers are generally referred to as time servers, and we canconfigure a Cisco router to get its time from a time server.

It's strongly recommended that your network's "outside" router receive itstime from a public NTP timeserver. For the latest IP addresses of theseservers, just do a quick search on the termpublic NTP servers.

T-shooting tip-in: NTP uses UDP port 123, so don't block that port with anACL.

Cisco routers can serve as an NTP server, client, orpeer. They can alsodepend on NTP broadcasts for the correct time.

As you'd expect, an NTP Server gives Clients the correct time. Clientaccept this time synchronization message and set their internal clockaccordingly. Clients will notsend NTP time synch messages back to theServer.

We're not limited to the traditional Server/Client relationship. NTP Peerssend NTP messages to each other, and either peer can send time synchmessages to the other.


19/30

If we configure NTP to run in broadcast mode or multicast mode, theServer will broadcast / multicast its NTP messages to the Clients. Keep inmind that a router will not forward broadcasts or multicasts by default, andthat includes NTP messages.

Setting The Clock And Configuring NTP

It's highlyrecommended that you use an NTP public timeserver as yourNTP Master time source. If you use one of your network routers as anNTP Master, it's imperative that you use NTP authentication and/or ACLsto prevent routers outside your network from attempting to synchronizewith your router.

Since this is a lab environment, we'll configure R1 as our NTP Master.R1 will serve as R3's NTP Server, and then we'll configure an NTPpeering between R2 and R3. The router number serves as the last octet

of each IP address in this lab.

Let's check the clock on R1.

R1#show clock

*10:05:18.097 UTC Mon Mar 1 1993

Hmm, that's not good. Better set the clock!

R1#clock ?

set Set the time and date

R1#clock set ?

hh:mm:ss Current Time

R1#clock set 10:53:00 ?

Day of the month

MONTH Month of the year

R1#clock set 10:53:00 23 ?

MONTH Month of the year

R1#clock set 10:53:00 23 April ?

Year


20/30

R1#clock set 10:53:00 23 April 2011 ?

R1#clock set 10:53:00 23 April 2011

Verify with show clock:

R1#show clock

10:53:02.599 EST Wed Apr 23 2011

Here are our NTP options:

R1(config)#ntp ?

access-group Control NTP access

authenticate Authenticate time sources

authentication-key Authentication key for trusted time sources

broadcastdelay Estimated round-trip delay

clock-period Length of hardware clock tick

master Act as NTP master clock

max-associations Set maximum number of associations

peer Configure NTP peer

server Configure NTP server

source Configure interface for source address

trusted-key Key numbers for trusted time sources

We're going to configure R1 as the NTP Master for our network.

R1(config)#ntp master

On R3, the clock is set to 1993 as well.

R3#show clock

*10:12:50.193 UTC Mon Mar 1 1993

We'll configure R3 to use 172.12.13.1 as its NTP server.R3(config)#ntp server 172.12.13.1

Let's go to R2 and set up an NTP peering with R3, configuring ntp peeronR3 before heading for R2. This peering will not use authentication.

A few moments later, the peering is in place and R2 reflects the correcttime.

R2#show clock

15:04:51.918 UTC Wed Apr 23 2008

The commands show ntp status and show ntp associations will verify yourNTP configuration, The key phrase in show ntp status is "clock issynchronized".

R2#show ntp status

Clock is synchronized, stratum 10, reference is 172.12.23.3

nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is

2**19

reference time is CBB9CEC8.17FBD1B8 (15:05:44.093 UTC Wed Apr 23 2008)

clock offset is -0.6214 msec, root delay is 37.20 msec

root dispersion is 5.04 msec, peer dispersion is 0.53 msec


21/30

R2#show ntp associations

address ref clock st when poll reach delay offset

*~172.12.23.3 172.12.13.1 9 61 64 376 6.4 -

0.62 * master (synced), # master (unsynced), + selected, - candidate,

~ configured

Like BGP routes, it's not enough just to see the association in this table.

The asterisk next to 172.12.23.3 indicates the NTP time server that the

local router has synched up with. If you don't see at least one addresswith an asterisk next to it, there's a problem. (Remember to give NTPconfigs a few minutes to take effect and for synchronization to take placein real-world networks...sometimes more than a few minutes!)

If we're fortunate and smart enough to have NTP redundancy, a routermay have more than one Server to choose from, and you may prefer onespecific server over the other. Use thepreferoption at the end of the ntpservercommand to specify a preferred NTP server.

R3(config)#ntp server 172.12.13.1 ?

key Configure peer authentication key

prefer Prefer this peer when possible

source Interface for source address

version Configure NTP version

R3(config)#ntp server 172.12.13.1 prefer

Additional Clock Commands

At the beginning of this lab, I used the set clockcommand to, well, set the

clock! There are two other clock options you're advised to use to keepyour network time synched up - timezone and summer-time.

The timezone option adjusts the time for your local timezone. Here inVirginia in the United States, we're on Eastern Standard Time, so I'll setthe clock to adjust for that.

R1(config)#clock timezone ?

WORD name of time zone

R1(config)#clock timezone EST ?

Hours offset from UTC

R1(config)#clock timezone EST -5

To allow the router to adjust to Daylight Savings Time, use the summer-time option followed by your local time zone. Note that summer-timerequires the hyphen in the middle.

R1(config)#clock summertime ?

% Unrecognized command

R1(config)#clock summer-time ?

WORD name of time zone in summer

R1(config)#clock summer-time EST ?


22/30

date Configure absolute summer time

recurring Configure recurring summer time

R1(config)#clock summer-time EST recurring

Logging

By default, debug output and error messages are sent to the consoleterminal, but we can also send messages to a separate Syslog server, the

logging buffer, or the VTY lines.

You can see some logging defaults with show logging:

R1#show logging

Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0

flushes,

0 overruns)

Console logging: level debugging, 23 messages logged

Monitor logging: level debugging, 0 messages logged

Buffer logging: level debugging, 23 messages logged

Logging Exception size (4096 bytes)

Trap logging: level informational, 27 message lines logged

Log Buffer (4096 bytes):

00:00:08: %LINK-3-UPDOWN: Interface Ethernet0, changed state to up

00:00:08: %LINK-3-UPDOWN: Interface Serial0, changed state to up

00:00:08: %LINK-3-UPDOWN: Interface Serial1, changed state to down

00:00:09: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed

state

to up

One important default is the size of the log buffer - 4096 bytes. I didn't putthe entire contents of the log here, but it's vital to keep in mind that this

log is circular, meaning that when it gets full, the newest entries willoverwrite the oldest entries.

When using the logging feature during troubleshooting, it may be a goodidea to increase the size of the log.

IOS Help illustrates that we have more than a few options for logging.

R1(config)#logging ?

Hostname or A.B.C.D IP address of the logging host

buffered Set buffered logging parameters

console Set console logging levelexception Limit size of exception flush output

facility Facility parameter for syslog messages

history Configure syslog history table

host Set syslog server host name or IP address

monitor Set terminal line (monitor) logging level

on Enable logging to all supported destinations

rate-limit Set messages per second limit

source-interface Specify interface for source address in logging

transactions

trap Set syslog server logging level

We have to walk a fine line between making the log too large and not

making it large enough. One option for keeping the size of the log down


23/30

is limiting the severity of messages that are logged.

Specifying a severity level means that all messages that match thatseverity level and all levels below that will be logged. If we configuredlogging buffered 5here, all messages up to and including the Notificationslevel would be logged.

R1(config)#logging buffered ?

Logging severity level

Logging buffer sizealerts Immediate action needed (severity=1)

critical Critical conditions (severity=2)

debugging Debugging messages (severity=7)

emergencies System is unusable (severity=0)

errors Error conditions (severity=3)

informational Informational messages (severity=6)

notifications Normal but significant conditions (severity=5)

warnings Warning conditions (severity=4)

R1(config)#logging buffered 8194

You can also limit the severity of messages logged to the console with thelogging console command.

R1(config)#logging console ?

Logging severity level

alerts Immediate action needed (severity=1)

critical Critical conditions (severity=2)

debugging Debugging messages (severity=7)

emergencies System is unusable (severity=0)

errors Error conditions (severity=3)

guaranteed Guarantee console messages

informational Informational messages (severity=6)

notifications Normal but significant conditions (severity=5)

warnings Warning conditions (severity=4)

logging synchronous prevents debugs and log messages from interruptingyou while you're typing - instead, those messages are held until the routerdetects that you're not entering data, and the messages are shown then.

A really helpful command, since few things are more annoying thanhaving an extended ACL config interrupted by a logging message. Here'show you use the logging synchronous command for telnet sessions:

R1(config)#line vty 0 4

R1(config-line)#logging synchronous

Should you have to turn logging off, simply run no logging on.

R1(config)#no logging on

R1#show logging

Syslog logging: disabled

To enable logging again, run logging on.

R1(config)#logging on


24/30

R1#show logging

Syslog logging: enabled

Interestingly enough, turning logging off actually puts more of a load onthe router's resources than keeping it on!

When I'm using logging as a troubleshooting tool, I really like to have thelog entries timestamped. Surprisingly, this is not a default. Enable thiswith the service timestamps log uptime or service timestamps logdatetime, depending on your preference. I personally preferlog datetime.

R1(config)#service timestamps ?

debug Timestamp debug messages

log Timestamp log messages

R1(config)#service timestamps log ?

datetime Timestamp with date and time

uptime Timestamp with system uptime

R1(config)#service timestamps log datetime

You may also want to direct the log entries to a Syslog server. To do so,use the loggingcommand followed by the hostname or IP address of theSyslog server.

R1(config)#logging ?

Hostname or A.B.C.D IP address of the logging host

R1(config)#logging 172.1.1.1

If you change logging options on a client's router while t-shooting, be sureto change them back to the original settings before you leave.

Everyone remembers an admin who shows professionalism.

Everyone remembers the other kind, too.

If I Could Turn Back Time...

... I wouldn't have done that to my Cisco router config.

There's a way to get your old config back, though, whatever "that" was --the config replace and config rollbackfeatures.

While the command config replace will replace your running config with asaved (or "archived", in fancy talk) config file, this feature does requiresome advance planning.

This link takes you to Cisco's config guide for this feature. It's not requiredreading for the exam, but it wouldn't hurt to spend a few minutes with it.

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t7/feature/guide/gtrollbk.html


25/30

Of course, the old-fashioned method of using the copycommand to copythe running-config file to a TFTP server works, soo. As that guide notes,the copy method doesn't offer you all of the file management options thatthe configure replace feature does -- but sometimes, all you want is thatconfig.

Using both isn't a bad idea.

The copy command's simple, just remember the first option is whereyou're copying from...

R1#copy ?

/erase Erase destination file system.

flash: Copy from flash: file system

flh: Copy from flh: file system

ftp: Copy from ftp: file system

null: Copy from null: file system

nvram: Copy from nvram: file system

rcp: Copy from rcp: file system

running-config Copy from current system configuration

startup-config Copy from startup configuration

system: Copy from system: file system

tftp: Copy from tftp: file system

... and the second option is where you're copying to.

R1#copy running-config ?

flash: Copy to flash: file system

flh: Copy to flh: file system

ftp: Copy to ftp: file system

lex: Copy to lex: file system

null: Copy to null: file system

nvram: Copy to nvram: file system

rcp: Copy to rcp: file system

running-config Update (merge with) current system configurationstartup-config Copy to startup configuration

system: Copy to system: file system

tftp: Copy to tftp: file system

The Cisco Embedded Event Manager

The Tshoot exam is not an EEM config exam - although there could be aseparate exam just for this tool. It has a ton of features, so we'll hit just afew high points here and then I'll give you a link for further reading that'snon-essential for the exam, but is a good reference if you're going to useEEM.

Basically, EEM allows you to closely track your admins and what they doon the network - and for purposes of change control, perhaps preventthem from carrying out certain tasks.

If you're not familiar with the term change control - and even if you are -here's a great definition of this term from Wikipedia:

"Change control is a formal process used to ensure that changes to


26/30

a product or system are introduced in a controlled and coordinatedmanner. It reduces the possibility that unnecessary changes will beintroduced to a system without forethought, introducing faults into thesystem or undoing changes made by other users of software."

If you've ever worked with an admin who really flew by the seat of his/herpants, you'll understand the importance of change control and itsenforcement.

You can configure EEM to list some events and actions as permissible byall admins, have custom messages sent to the console after some actions("Hey! You! Document what you just did!" - or perhaps something a littlemore formal.) You can also have other actions prohibited by EEM forcertain admins, and have those admins sent a little reminder on thescreen about why their last command wasn'taccepted.

Some EEM links:

Cisco's EEM main page: http://bit.ly/dVXwFf

Wikipedia EEM Page: http://bit.ly/gnAJju

The Security Device Manager

When Cisco GUIs first hit, there was a hue and cry from some that thiswas the death knell for network admins, they could hire anyone off thestreet to admin Cisco routers and switches, etc etc etc.

That hasn't happened.

Just as with EEM, the TShoot exam is not an SDM configuration exam.It's a good idea to be familiar with it for the real world, though, andcertainly for your Security certification studies.

Rather than put just a few screen shots here, I'd rather give you a copy ofmy CCNA Security Study Guide in ebook format - there are severalhundred screen shots in there and you can see what it's all about. If youdidn't get one with your order of this ebook, send me a Tweet atccie12933 and I'll get you set up with one. No emails on this one, please.

Thanks!

NBAR

The Network Based Application Recognition feature is a great place tostart your QoS deployment - after all, before you worry about marking orclassifying traffic, you've got to identify the traffic on your network! NBARworks from OSI layers 4 - 7 to identify and classify protocols, making itpossible for you to create an accurate QoS configuration.

You may also be familiar with the use of NBAR to block the "Code Red"

virus. It's highly doubtful you'll see any questions on your exam regarding


27/30

that, but if you'd like to learn more about that, just put "nbar code red" inyour favorite search engine and you'll quickly find some Ciscodocumentation on that subject.

NBAR has two basic purposes:

Identifying traffic on a per-protocol basis Application monitoring

NBAR performs application monitoring via its Protocol Discoveryfeature. Protocol Discovery is enabled on a per-interface basis, and thishelpful feature can tell you what applications are running on that interfaceand the amount of bandwidth they're using.

NBAR is capable of categorizing traffic in three ways:

protocol port number

payload content (default limit: 400 bytes)

NBAR also examines network traffic and classifies it by TCP and UDPport numbers, whether those port numbers be statically or dynamicallyassigned. When dynamic port numbers are involved, NBAR is said to beperforming a stateful inspection.

NBAR can identify IP protocols that are neither TCP-based not UDP-based as well, and NBAR's capabilities go beyond identifying portnumbers. NBAR can identify and classify WWW traffic according to theURL as well.

Best of all, NBAR's capabilities are continually extended through thedevelopment of Packet Description Language Modules (PDLM). Notonly do these PDLMs allow your NBAR deployment to identify more andmore different types of traffic, but a router reload is not necessary, andyou don't need a new IOS image.

We'll look at the command to load PDLMs in just a moment, but for nowkeep in mind that you're generally going to store PDLMs in Flash.

You will need a Cisco CCO number to access PDLMs; the download pagecan be quickly accessed through your favorite search engine.

NBAR can go above and beyond "just" examining port numbers. Throughsubport classification, NBAR can examine the payload and classifypackets on other values, such as the Multipurpose Internet Mail Extension(MIME) type or, as previously mentioned, the URL. This detailed packetexamination is called deep packet inspection. (NBAR cannot supportmore than 24 MIME or URL matches at the same time.)

NBAR Configuration


28/30

Cisco Express Forwarding (CEF) must be enabled on the interfaces thatwill run NBAR.

Enabling NBAR protocol discovery (an optional command, but required forapplication monitoring):

R2(config-if)#ip nbar protocol-discovery

Verify with show ip nbar protocol-discovery. To illustrate the results, I ran

BGP on this interface for a few minutes before running this command.This command's output is verbose to say the least, so I will not show theentire output here.

R2#show ip nbar protocol-discovery

Serial0/0

Input Output

----- ------

Protocol Packet Count Packet Count

Byte Count Byte Count

5min Bit Rate (bps) 5min Bit Rate (bps)

5min Max Bit Rate (bps) 5min Max Bit Rate (bps)

bgp 19 38

1197 2033

0 0

0 0

Loading a PDLM (assuming the PDLM is in Flash) :

R2(config)#ip nbar pdlm ?

WORD Full path of the PDLM file

R2(config)#ip nbar pdlm flash://ccnp.pdlm

To verify the port numbers used by NBAR, run show ip nbar port-map. Toverify a single port number, put the protocol name at the end of thatcommand. I've truncated the output of the first command.

R2#show ip nbar port-map

port-map bgp udp 179

port-map bgp tcp 179

port-map citrix udp 1604

port-map citrix tcp 1494port-map cuseeme udp 7648 7649 24032

port-map cuseeme tcp 7648 7649

R2#show ip nbar port-map dns

port-map dns udp 53

port-map dns tcp 53

NBAR Limitations And Restrictions

If you're interested in using NBAR in your network, be sure to visit Cisco's


29/30

website for the latest documentation. In the meantime, here's a partial listof NBAR's limitations.

NBAR does *not* support or analyze:

Non-IP traffic MPLS packets Fragments Packets created by the local router (the one actually running NBAR) Packets destined for the local router

Additionally, you cannot run NBAR on these interface types:

Interfaces running encryption or tunneling Dialer interfaces Fast Etherchannel (NOTE - that's Fast Etherchannel, not Fast

Ethernet)

The dialer interface limitation was removed as of IOS 12.2(4)T.

And one more thing...

The only switching mode that supports NBAR is CEF.

Intro To AutoQOS

AutoQoS dynamically discovers what applications are running on yournetwork - with the help of NBAR - and creates the appropriate QoSdeployment.

AutoQoS is Cisco's attempt to make configuring QoS easier in today'scomplex networks. Several of their documents mention a cost savings aswell, since this app makes it possible for someone without in-depthknowledge of QoS to successfully deploy QoS in their network.

The phrase "without calling a consulting firm and paying a consultant" isleft unspoken.

It's hard foranyone to think of everything a QoS scheme needs, even anetwork admin, but look at just a partial list of what AutoQoS will take care

of for us.

Autoconfiguration of LLQ for voice traffic, as well as guaranteeing acertain level of bandwidth to that voice traffic.

Dynamically configures and enables RTP Header Compression(cRTP), as well as link fragmentation and interleaving (LFI)

Autoconfigures traffic shaping as needed in compliance with Ciscobest practice

Overall, supports Frame, ATP, PPP, and HDLC - but there arecertain interface types that do not run AutoQoS.


30/30

We've got quite a few prerequisites to concern ourselves with, along withone thing we can'thave running on the interface...

AutoQoS Prerequisites:

An IP address must be configured on low-speed interfaces (768 kbpsor slower)

bandwidth command is required on interfaces on both ends link, andthey should be set to the same value

CEF must be enabled at the interface level NBAR will be used to identify applications and traffic

Here's what must NOT be configured:

QoS policies must be removed from the interface(s) that will runAutoQoS. If there is a policy present, AutoQoS will not work

To be blunt, almost every AutoQoS issue you run into will involve theabove.

The overall config has two steps:

Collect the info

Allow AutoQoS to use that info to create a QoS scheme

We collect the info with auto discovery qos on the interface, and let it"cook" for a while.

R1(config)#int s0/0

R1(config-if)#auto discovery qos

Obviously, a few seconds isn't long enough to cook, as show autodiscovery qos shows us.

R1#show auto discovery qos

Serial0/0

AutoQoS Discovery enabled for applications

Discovery up time: 18 seconds

AutoQoS Class information:

No AutoQoS data discovered

Once you do have info to work with, just run the auto qos command onthe interface to allow the router to come up with and apply an appropriatepolicy.

R1(config-if)#auto qos

No AutoQoS data discovered

Copyright 2011 The Bryant Advantage, inc. All Rights Reserved.

1-tshooting models, methods, tools and tips

Documents