7750 sr series troubleshooting guide

Alcatel

This document contains Confidential Information of Alcatel.

31NAN0090

Issue Version 2.0, Aug. 5th, 2004

7750 SR Series Troubleshooting Guide

Application Note

IPD Support & Services

Abstract: This document provides detailed information on diagnosing faults in R2.0 of the 7750 SR

Alcatel 31NAN0090 – 7750 Troubleshooting Guide

2

Table of contents

1. INTRODUCTION ...............................................................................................................................................................5

1.1. INTENDED AUDIENCE FOR THIS GUIDE .........................................................................................................................5 1.2. HOW THIS DOCUMENT IS ORGANIZED...........................................................................................................................5 1.3. WHERE TO BEGIN?........................................................................................................................................................6 1.4. RELATED DOCUMENTS .................................................................................................................................................6

2. TROUBLESHOOTING PROCESS...................................................................................................................................7

2.1. ESTABLISHING A BASELINE ..........................................................................................................................................7 2.2. CHARACTERIZE THE PROBLEM......................................................................................................................................8 2.3. IDENTIFY THE ROOT CAUSE..........................................................................................................................................9 2.4. PLAN YOUR ACTIONS & RESOLVE THE PROBLEM........................................................................................................10 2.5. VERIFY SOLUTIONS.....................................................................................................................................................11

3. TROUBLE SHOOTING TOOLS ....................................................................................................................................12

3.1. EVENT LOGS ...............................................................................................................................................................12 3.1.1. Event logging overview .........................................................................................................................................12

3.1.1.1 Event Sources ................................................................................................................................................................ 13 3.1.1.2 Event Control ................................................................................................................................................................ 14 3.1.1.3 Log manager.................................................................................................................................................................. 17 3.1.1.4 Event Filter Policies ...................................................................................................................................................... 17 3.1.1.5 Log Destinations............................................................................................................................................................ 19

3.1.2. List of show commands for event logging .............................................................................................................22 3.2. SERVICE MIRRORING...................................................................................................................................................23

3.2.1. Service mirroring overview ...................................................................................................................................23 3.2.2. Mirror implementation ..........................................................................................................................................24

3.2.2.1 Mirror Source and Destinations..................................................................................................................................... 25 3.2.2.2 Mirroring performance .................................................................................................................................................. 27

3.2.3. Mirroring configuration........................................................................................................................................27 3.2.3.1 Mirror configuration process overview ......................................................................................................................... 29 3.2.3.2 Mirror configuration components.................................................................................................................................. 29 3.2.3.3 Basic mirror configuration Example.............................................................................................................................. 30 3.2.3.4 Mirror configuration Notes............................................................................................................................................ 33 3.2.3.5 List of CLI commands to configure Mirroring parameters............................................................................................ 34

3.3. OA&M COMMANDS FOR TROUBLESHOOTING .............................................................................................................36 3.3.1. LSP Diagnostics ....................................................................................................................................................36 3.3.2. SDP Diagnostics ...................................................................................................................................................36 3.3.3. Service Diagnostics ...............................................................................................................................................37 3.3.4. VPLS MAC Diagnostics ........................................................................................................................................38 3.3.5. OAM Command Summary.....................................................................................................................................40

4. HARDWARE OPERATIONAL STATUS......................................................................................................................42

4.1. 7750 SR-12 HARDWARE OVERVIEW...........................................................................................................................42 4.2. VERIFYING ROUTER BOOT SEQUENCE........................................................................................................................45 4.3. VERIFYING MANAGEMENT CONNECTION OPERATIONAL STATUS...............................................................................45

4.3.1. Console Port Management Connection.................................................................................................................45 4.3.2. Telnet Management Connection............................................................................................................................46

4.4. VERIFYING CHASSIS OPERATIONAL STATUS...............................................................................................................46 4.4.1. Chassis Configurations .........................................................................................................................................46 4.4.2. Things to Check - Power Supply ...........................................................................................................................48 4.4.3. Things to Check - Fans..........................................................................................................................................49

4.5. VERIFYING SF/CPM OPERATIONAL STATUS ..............................................................................................................50 4.5.1. Minimum Configuration ........................................................................................................................................50 4.5.2. SF/CPM LED Status..............................................................................................................................................50 4.5.3. CLI commands for SF/CPM troubleshooting........................................................................................................51 4.5.4. CLI commands for SF/CPM health check .............................................................................................................53

4.6. VERIFYING IOM OPERATIONAL STATUS ....................................................................................................................57 4.7. VERIFYING MDA OPERATIONAL STATUS...................................................................................................................58

31NAN0090 – 7750 Troubleshooting Guide Alcatel

3

5. SYSTEM LEVEL CONFIGURATION VERIFICATION............................................................................................60

5.1. SUMMARY OF SYSTEM CONFIGURATION VERIFICATION...............................................................................................60 5.2. SYSTEM INITIALIZATION TROUBLESHOOTING .............................................................................................................61

5.2.1. Boot Option File configuration .............................................................................................................................61 5.2.2. Troubleshooting notes on BOF configuration.......................................................................................................63 5.2.3. Commands to check config file contents ...............................................................................................................63

5.3. VERIFY SYSTEM MANAGEMENT CONFIGURATION .......................................................................................................67 5.3.1. Display system information ...................................................................................................................................68 5.3.2. Verify Synchronization and Redundancy...............................................................................................................69 5.3.3. Verify timing configuration ...................................................................................................................................70 5.3.4. Verify SNTP configuration ....................................................................................................................................72

5.4. SECURITY ACCESS CONFIGURATION ...........................................................................................................................72 5.4.1. Authentication, Authorization and Accounting .....................................................................................................72 5.4.2. How AAA is configured .........................................................................................................................................74 5.4.3. Security Configuration Components .....................................................................................................................76

5.4.3.1 Configuring Management access filters......................................................................................................................... 76 5.4.3.2 Configuring Password management parameters............................................................................................................ 77 5.4.3.3 Configuring profiles ...................................................................................................................................................... 78 5.4.3.4 Configuring User access parameters.............................................................................................................................. 79 5.4.3.5 Configuring RADIUS Authentication ........................................................................................................................... 80 5.4.3.6 Configuring RADIUS Authorization............................................................................................................................. 81 5.4.3.7 Configuring VSA when RADIUS Authorization is enabled ......................................................................................... 82 5.4.3.8 Configuring RADIUS Accounting ................................................................................................................................ 85 5.4.3.9 Enabling TACACS+ Authentication ............................................................................................................................. 86 5.4.3.10 Configuring TACACS+ Authorization.......................................................................................................................... 87 5.4.3.11 Configuring TACACS+ Accounting ............................................................................................................................. 87 5.4.3.12 Enabling SSH ................................................................................................................................................................ 88 5.4.3.13 Configuring Login controls ........................................................................................................................................... 88

5.4.4. SNMP security configuration ................................................................................................................................89 5.4.4.1 SNMP overview ............................................................................................................................................................ 89 5.4.4.2 Which SNMP version to use.......................................................................................................................................... 92 5.4.4.3 SNMP security configuration components .................................................................................................................... 93 5.4.4.4 Commands displaying SNMP security configuration.................................................................................................... 94

5.4.5. User Access failure troubleshooting .....................................................................................................................94 5.5. VERIFY EVENT & ACCOUNTING LOGS CONFIGURATION .............................................................................................95

5.5.1. Accounting logging Overview ...............................................................................................................................95 5.5.2. Verifying the logging configurations.....................................................................................................................98

6. COMMON TROUBLESHOOTING SCENARIOS .....................................................................................................100

6.1. LAYER 1 & LAYER 2 PROBLEMS ............................................................................................................................... 100 6.1.1. How to show Layer 1 & Layer 2 alarms .............................................................................................................100 6.1.2. Verify cards, MDAs and ports configuration ......................................................................................................100 6.1.3. How to show or clear statistics on a port or a LAG or a SAP.............................................................................101 6.1.4. How to show or modify the operational status of a port .....................................................................................102 6.1.5. How to loop ports................................................................................................................................................102

6.2. OSPF PROBLEMS......................................................................................................................................................104 6.2.1. Commands common to any OSPF troubleshooting.............................................................................................104 6.2.2. OSPF not come up...............................................................................................................................................106

6.3. BGP PROBLEMS........................................................................................................................................................111 6.3.1. Commands common to any BGP troubleshooting...............................................................................................111 6.3.2. BGP peer session not established........................................................................................................................112 6.3.3. BGP load balancing issue ...................................................................................................................................115

6.4. PREFIX-LIST (ACCESS-LIST) IN THE ROUTE POLICY ..................................................................................................117 6.5. BLACK HOLING PROBLEMS .......................................................................................................................................120 6.6. LDP NOT ESTABLISHED ............................................................................................................................................121 6.7. CPU UTILIZATION HIGH SCENARIO ..........................................................................................................................122 6.8. TROUBLESHOOTING IES (INTERNET ENHANCED SERVICE) SERVICES .......................................................................123 6.9. NETWORK MONITORING ...........................................................................................................................................125

7. MISCELLANEOUS ........................................................................................................................................................128


4

TABLES Table 1: Event Severity Levels .................................................................................................................. 14 Table 2: Valid Filter Policy Operators .......................................................................................................... 18 Table 3: 7750 SR OS to Syslog Severity Level Mappings ........................................................................... 22 Table 4: CLI Commands to Configure Mirroring Parameters ...................................................................... 35 Table 5: Chassis Front View Features .......................................................................................................... 43 Table 6: Chassis Rear View Features ........................................................................................................... 44 Table 7: Console Configuration Parameter Values....................................................................................... 46 Table 8: 7750 SR-12 Hardware Component Operating Requirements ......................................................... 47 Table 9: 7750 SR-12 AC Power Supply LED Descriptions ......................................................................... 48 Table 10: SF/CPM Field Descriptions .......................................................................................................... 51 Table 11: Index of system configuration verification tasks .......................................................................... 60 Table 12: Configuring Authentication .......................................................................................................... 74 Table 13: Configuring Authorization............................................................................................................ 75 Table 14: Configuring Accounting ............................................................................................................... 76 Table 15: Accounting Record Name and Collection Periods ................................................................ 97

FIGURES: Figure 1: Event Logging Block Diagram...................................................................................................... 12 Figure 2: show log application command output .......................................................................................... 14 Figure 3: Service Mirroring ......................................................................................................................... 24 Figure 4: Local mirroring Example .............................................................................................................. 28 Figure 5: Remote mirroring Example ........................................................................................................... 29 Figure 6: Service mirror configuration and implementation flow ................................................................ 29 Figure 7: Local Service Mirroring Configuration......................................................................................... 31 Figure 8: Remote Service Mirroring Configuration...................................................................................... 32 Figure 9: 7750 SR-12 Chassis Front View .................................................................................................. 43 Figure 10: 7750 SR-12 Chassis Rear View.................................................................................................. 44 Figure 11: Management Console Port Connection ...................................................................................... 45 Figure 12: Telnet Management Port Connection ......................................................................................... 46 Figure 13: 7750 SR-12 AC Power Supply LEDs......................................................................................... 48 Figure 14: SF/CPM Front Panel .................................................................................................................. 50 Figure 15: SNMPv1 and SNMPv2c Configuration and Implementation Flow .................................... 93 Figure 16: SNMP Configuration Components ......................................................................................... 93 Figure 17: Alarm relationships on the 5620 SAM GUI .............................................................................. 127


5

1. Introduction

1.1. Intended Audience for this Guide

This document has been written to address the needs of network administrators and network support personnel who are on the front-line of diagnosing issues with the Alcatel 7750 SR. Typically, this includes network operations groups within customer organizations, Alcatel 2nd Line Support, various Technical Assistance Center (TAC) staff, sales engineers and pre-sales engineers. This guide requires knowledge of IP networking technology.

1.2. How This Document is organized

This Guide provides an overview of the troubleshooting process and provides a convenient description of all the troubleshooting tools that are available on the Alcatel 7750 SR. The Guide then breaks down troubleshooting by the major hardware components of the router in addition to providing guidance to troubleshooting system level, router level and service level configuration issues.

• Troubleshooting Process provides a systematic approach to troubleshooting router problems

that is based on the categorization of the symptoms of the trouble, the collection of descriptive information related to the problem, the analysis of the information to identify potential causes and the resolution through a systemic application of corrective actions.

• Troubleshooting Tools describes the tools and utilities that are used to configure, monitor and troubleshoot the Alcatel 7750 SR.

• Hardware Operational Status describes how to verify the operational status and validate the configuration of the hardware components of the Alcatel 7750 SR:

o SF/CPM

o IOM

o MDA

• System Level Configuration Verification describes how to verify the proper configuration of system components such as the Boot Option File, the System Management settings, the router security settings and the system settings for the hardware components of the Alcatel 7750 SR.

• Common Troubleshooting Scenarios provides information on troubleshooting problems that commonly occur at layer 1& layer 2 (such as IOM, MDA or port level), router level (such as OSPF, BGP or route policy), and other specific scenarios.


6

1.3. Where to Begin?

There are many and various methodologies that are followed to troubleshoot problems, be that a problem in a network, in a computer, in an application, or even in a car. All methodologies will invariably have the same or at least similar actions and goals, these being to identify, characterize and finally resolve the problem.

After having established a baseline, the 1st step in troubleshooting any node is to start in the "Event Logs" - where the alarms are logged. The Event logs maybe stored locally on the node or remotely on a server or on the Alcatel 5620 SAM. Collect all the symptoms you can for the problem node as the more information you have to work from, the easier it is to isolate the cause and figure out how to resolve the problem. Other information you will probably want to collect includes hardware, software and nodal configuration information, equipment and service operating statistics and service specific configuration data.

More detail is on the troubleshooting process is provided in section 2 Troubleshooting Process.

This guide is based on the hardware and software introduced in the Alcatel 7750 SR R2.0.

1.4. Related Documents

Please refer to the following for further information on the Alcatel 7750 SR:

5620 SRM r1.2 New Feature Training (Service Assurance) - 07NPT0067.E_(Service Assurance)_v1.1.ppt

Alcatel 5620 Service Router Manager R2.0 User Guide - 5620SRM20_UG.pdf

Alcatel 7750 SR-12 Installation Guide - 7750_SR-12_Installation_Guide_Rev-02.pdf

Alcatel 7750 SR OS System Guide - 7750_SR_OS_System_Guide_2.0.pdf

Alcatel 7750 SR OS Services Guide - 7750_SR_OS_Services_Guide_2.0.pdf

Alcatel 5620 SAM Service Aware Manager R2.0 General Information Book

Note: The Alcatel 5620 SRM is now known as the Alcatel 5620 SAM


7

2. Troubleshooting Process

Troubleshooting and problem solving is basically the same thing. In either case, there is the acknowledgment that something in the network, be that a component of the network or a service within the network, is not operating within expected operating parameters. The problem can result in a total or catastrophic failure in the network, or the problem can manifest itself intermittently, or then again, the problem might have resulted in degradation of how the service is performing.

There are many accepted methodologies for troubleshooting a problem and they all must naturally start with the identification that a problem exists. This implies a certain level of understanding of the designed state and behavior of a network and the services that are using that network as well as an identification of a symptom that the desired behavior is no longer there. This identification can come in the form of an alarm received from a network component, through the analysis of network capacity and performance data or even from a call from a customer reporting a problem with their service.

The basis for effective troubleshooting is in having a well understood baseline for the network and services, a detailed knowledge of the elements of the network, from transport to routing, a thorough understanding of the services and how they operate, and finally, a degree of expertise in the use of troubleshooting tools that are available in the network elements and the network management systems. These elements are discussed in more details in the following sections of this guide.

2.1. Establishing a Baseline

Having a thorough knowledge of your network and how it functions under normal conditions is essential if you want to be efficient in troubleshooting problems as it allows for rapid and easy identification that a fault exists in your network. It is therefore essential that a sound baseline of your network and services be established and rigorously maintained since a network is never a static environment. Customer churn, new service introductions, new service points of presence are added, links fail, etc…

How detailed should that baseline be? That depends on how much time and money you want to invest in establishing the baseline, on the level of expertise and degree of experience your operations staff has and on how good the fault management capabilities are in your network management system. Establishing a baseline typically includes:

• Creating Network Configuration Documentation

• Create End-System Networking Configuration Documentation

• Periodically backing up router running configurations

• Storing the backups at a safe, off-site location

• Documenting service descriptions and service SLAs

• Collecting and understanding statistics on traffic flows, router and trunk utilization levels

• Document customer profiles, customer contact numbers


8

• Document the General Troubleshooting Process

Maintaining a detailed history of problems, their symptoms, how the root cause was identified and how the problem was resolved is also a powerful tool towards efficient troubleshooting. Your problem tracking system should maintain a history of network and service problems and their resolution and include details such as:

• Problem symptoms

• Associated alarms and network event messages

• Network conditions, such as link failures, congestion, packet discards

• Type, version and configuration of hardware and software for the affected network elements

• Description of service impacts

• Results of any corrective actions

• Problem resolution

2.2. Characterize the Problem

A computer network, such as the Internet, is considered to be a well defined system whose state and expected behavior can be well defined and documented. The goal in troubleshooting well defined systems is to return the system to the as-designed behavior state. The first step in returning the system to its design intent is to fully characterize the problem state.

Part of characterizing problems is differentiating between total failures and problems that result in a degradation in performance. For a customer that has a single DS3 link into the network, a failure of the access router results in a total failure for that customer. A core router operating above 80% average utilization will start to discard packets which will result in a degradation of performance for at least certain applications running through that router. Performance degradations will exhibit greatly different symptoms from total failures and may not generate alarms or significant network events.

Multiple problems can and often will happen at the same time and can manifest same, related or completely different symptoms. It is therefore critical when identifying symptoms that as many characterizing parameters be collected from the network as possible including:

• Alarm files

• Error logs

• Network statistics

• Network analyzer traces

• Core dumps


9

• Serial line traces

• Stack dumps

• Output of various show commands in CLI (current configuration)

• Accounting logs

• Customer trouble reports

The more detailed the documented symptoms, the easier it is to identify the root cause of the problem. It is important to remember that in many cases the individual or the team that is recording the problem symptoms may not be the same people who will be finding the root cause and resolving the problem, therefore close attention to detail in recording the problem symptoms is crucial to rapid problem resolution.

Alarms can be viewed directly from the 7750 SR node alarm file or through the use of the fault management features available in the 5620 SAM. The 5620 SAM converts SNMP traps from network routers to events and alarms which can be easily correlated against the appropriate managed equipment and configured services and policies.

Some questions to answer and conditions to investigate when characterizing the problem are:

• Is it an intermittent problem, or is the problem static in nature?

• If the problem is intermittent, how often has it happened, is there a pattern?

• What alarms or network events are associated with the problem?

• Can you identify any congestion in routers or network links?

• Identify and record any changes that have taken place since the network was last functioning properly.

2.3. Identify the Root Cause

As mentioned, a particular symptom can be the result of more than one network problem. Successfully troubleshooting a problem state therefore involves the identification of the root cause of each and every individual cause of the problem state. It is entirely possible to fix the problem by trying a variety of actions, such as resetting a network link, rebooting a router, reseating an IO module, in the general case the intended solution will be arrived at more rapidly by following a systematic approach to troubleshooting. A systematic approach to identifying the root cause of the problem includes the following elements:

• Once the symptoms have been identified and thoroughly documented, first try to identify if they have anything in common and focus on the common stuff first and work out from there.


10

• Alarms available through the 5620 SAM contain vendor-specific and X.733 standardized probable cause that can be very useful in identifying the root cause.

• Statistics on alarms available from the 5620 SAM tell you how often an alarm has been raised based on specified scenarios that can be helpful in identifying the root cause of a problem.

• If the symptoms are present in different areas of the network try to identify what is common across these areas.

• Work on one problem at a time, fix that problem, then move on to the next.

• Divide the problem space into natural segments and try to isolate the problem to one of the segments. One way of segmenting the network is:

o LAN switching (edge access).

o LAN routing (distribution, core).

o Metropolitan-area networks.

o WAN (national backbone).

o Partner services (extranet).

o Remote access services.

• Try to determine the precise network state that existed before the problem appeared.

• Identify which specific functions are not working properly and focus on those.

• Extrapolate from the network alarms and network events what conditions could result in the observed symptoms. Test for these to see if the problem can be reproduced.

2.4. Plan your actions & Resolve the Problem

The actions you take will depend on the type of problem that you are trying to resolve. Critical problems that are affecting a wide range of services for a large number of gold service level customers require a different tact from minor problems affecting a small number of best-effort service customers. The former situation will by necessity require drastic and immediate actions to restore service while the latter can afford to take a little more time to ensure that the actions will not put any other services at risk. The key is to balance the risk of creating further service interruptions while attempting to restore service in the shortest possible timeframe. Whatever corrective action is planned, you should:

• Reproduce the symptom

• Document each step of the corrective action

• Test the corrective action


11

• Use CLI to verify behavior changes for each step

The next step after testing your hypothesis and verifying that the corrective action is going to correct the problem and not introduce any new symptoms is to apply the corrective action to the live network. When doing so, it is recommended to resolve the easiest problem, in terms of risk, effort and time, first.

2.5. Verify Solutions

After having taken corrective action to resolve the problem it is important to verify that the changes have not introduced new symptoms and that the original problem has been completed corrected. If new symptoms are detected or if the problem has only been mitigated, you need to start the troubleshooting process again.


12

3. Trouble shooting tools

3.1. Event logs

Event logs are the means of recording system generated events for later analysis. Should there be a fault within a 7750 SR system, event logs are the means for troubleshooting. Events are messages generated by the system by applications or processes within the 7750 SR.

3.1.1. Event logging overview

7750 SR OS supports event logging. Event logging controls the generation, dissemination and recording of system events for monitoring status and troubleshooting faults within the system. The logging:

• Provides you with logging information for monitoring and troubleshooting. • Allows you to select the types of logging information to be recorded. • Allows you to assign a severity to the log messages. • Allows you to select the source and destination of logging information.

Event logs are the means of recording system generated events for later analysis. Events are messages generated by the system by applications or processes within the 7750 SR. Figure 1 depicts a function block diagram of event logging.

Figure 1: Event Logging Block Diagram


13

3.1.1.1 Event Sources

The event sources are the main categories of events that feed the log manager. The 7750 SR groups events into four major categories.

• Security events - Events that pertain to attempts to breach system security. The security event source is all events that affect attempts to breach system security such as failed login attempts, attempts to access MIB tables to which the user is not granted access or attempts to enter a branch of the CLI to which access has not been granted. Security events are generated by the SECURITY application.

• Change events - Events that pertain to the configuration and operation of the node. The

change activity event source is all events that directly affect the configuration or operation of the node. Change events are generated by the USER application.

• Debug-trace events - Debug and trace messages that have been enabled for applications or

processes. The debug event source is all debugging and trace messages that have been enabled on the system. Debug events are generated by the DEBUG application.

• Main events - Events that pertain to 7750 SR OS applications that are not assigned to other

event categories/sources. Examples of applications within 7750 SR OS include IP, MPLS, OSPF, CLI, services, etc. Figure 2 displays the show log applications command output which displays all applications.


14

Figure 2: show log application command output

3.1.1.2 Event Control

Event control pre-processes the events generated by applications before the event is passed into the main event stream. Event control assigns the severity for each application event and whether the event should be generated or suppressed. The severity numbers and severity names supported in 7750 SR OS conform to ITU standards M.3100 X.733 & X.21 and are listed in Table 1.

Table 1: Event Severity Levels


15

Events that are suppressed by event control will not generate any event log entries as it never reaches the log manager. Event control maintains a count of the number of events generated (logged) and dropped (suppressed) for each application event. The severity of an application event can be configured in event control. Application events contain an event number and description that explains why the event is generated. The event number is unique within an application, but the number can be duplicated in other applications. The following example, generated by querying event control for application events, displays a partial list of event numbers and names.


16


17

3.1.1.3 Log manager

Events that are forwarded by event control are sent to the log manager. The log manager manages the event logs in the system and the relationships between the log sources, event logs and log destinations, and log filter policies. An event log has the following properties:

• A unique log ID The log ID is a short, numeric identifier for the event log. • One or more log sources The source stream or streams to be sent to log destination can be specified. The source must be identified before the destination can be specified. The events can be from the main event stream, events in the security event stream, events in the user activity stream, or all debug-trace messages in the debug stream. • One event log destination A log can only have a single destination. The destination for the log ID destination can be one of console, session, syslog, snmp-trap-group, memory, or a file on the local file system. • An optional event filter policy

An event filter policy defines whether to forward or drop an event or trap based on match criteria.

3.1.1.4 Event Filter Policies

The log manager uses event filter policies to allow fine control over which events are forwarded or dropped based on various criteria. Filter policies have a default action. The default actions are to either:

• Forward • Drop

Filter policies also include a number of filter policy entries that are identified with an entry ID and define specific match criteria and a forward or drop action for the match criteria. Each entry contains a combination of matching criteria that define the application, event number, severity, and subject conditions. The entry’s action determines how the packets should be treated if they have met the match criteria. Entries are evaluated in order from the lowest to the highest entry ID. The first matching event is subject to the forward or drop action for that entry.


18

Valid operators are displayed in Table 2:

Table 2: Valid Filter Policy Operators

A match criteria entry can include combinations of:

• Equal to or not equal to a given system application. • Equal to, not equal to, less than, less than or equal to, greater than or greater than or equal

to an event number within the application. • Equal to, not equal to, less than, less than or equal to, greater than or greater than or equal

to a severity level. • Equal to or not equal to an event subject string.

The following example shows the event filter policies configured on a 7750 SR.


19

3.1.1.5 Log Destinations

An event log within 7750 SR OS associates the event sources with logging destination. 7750 SR OS supports the following log destinations:

• Console

• Session

• Memory logs

• Log files

• SNMP trap group

• Syslog Only a single log destination can be associated with an event log or with an accounting log. An event log can be associated with multiple event sources, but it can only have a single log destination. A file destination is the only type of log destination that can be configured for an accounting log. Console

Sending events to a console destination means the message will be sent to all active console sessions. If there are no active console sessions, the event log entries are dropped. The console device can be used as an event log destination. Session

A session destination is a temporary log destination which directs entries to the active console session for the duration of the console session. When the session is terminated, the event log is removed. Event logs with a session destination are not stored in the configuration file. Event logs can direct log entries to the session destination.


20

Memory Logs

A memory log is a circular buffer. When the log is full, the oldest entry in the log is replaced with the new entry. When a memory log is created, the specific number of entries it can hold can be specified, otherwise it will assume a default value. An event log can send entries to a memory log destination. Default System Log

Log 99 is a pre-configured memory-based log which logs from the main event source (not security, debug/trace, etc.). Log 99 exists by default. The following example displays the log 99 configuration.

Log Files

Log files are stored on the compact flash devices (specifically cf1 or cf2) in the 7750 SR file system. A log file is identified with a single log file ID, but a log file will generally be composed of a number individual files in the file system. A log file is configured with a rollover parameter which determines how long in minutes an individual file which is a component of the log file should be written to before a new file is created for the log file ID. The retention time for a log file specifies the amount of time the file should be retained on the system based on the creation date and time of the file. The retention time is used as a factor to determine which files should be deleted first if the file system device nears 100% usage. One log file can only be attached to one log ID. When a log file is created, only the compact flash device for the log file is specified. Log files are created in specific subdirectories with standardized names depending on the type of information stored in the log file. Event log files are always created in the \log directory on the specified compact flash device.


21

SNMP Trap Group

An event log can be configured to send events to SNMP trap receivers by specifying an SNMP trap group destination. An SNMP trap group can have multiple trap-receivers with different trap destinations. Each trap receiver can have different operational parameters. A trap destination has the following properties:

• The IP address of the trap receiver. • The UDP port used to send the SNMP trap. • SNMP version (v1, v2c, or v3) used to format the SNMP notification. • SNMP community name for SNMPv1 and SNMPv2c receivers. • Security name and level for SNMPv3 trap receivers.

For SNMP traps that will be sent out-of-band through the Management Ethernet port on the SF/ CPM, the source IP address of the trap is the IP interface address defined on the Management Ethernet port. For SNMP traps that will be sent in-band, the source IP address of the trap is the system IP address of the 7750 SR. Each trap destination of a trap group receives the identical sequence of events as defined by the log ID and the associated sources and log filter applied. Syslog

An event log can be configured to send events to one syslog destination. Syslog destinations have the following properties:

• Syslog server IP address. • The UDP port used to send the syslog message. • The Syslog Facility Code (0 - 23) (default 23 - local7). • The Syslog Severity Threshold (0 - 7) - events exceeding the configured level will be sent.

Because syslog uses eight severity levels whereas the 7750 SR OS uses six internal severity levels, the 7750 SR OS severity levels are mapped to syslog severities. Table 3 displays the 7750 SR OS severity level mappings to syslog severities.


22

Table 3: 7750 SR OS to Syslog Severity Level Mappings

3.1.2. List of show commands for event logging

Information to view show commands Displays a list of all application names that can be used in event-control and filter commands.

show log applications

Displays event control settings for events including whether the event is suppressed or generated and the severity level for the event.

show log event-control [application [event-name | event-number]]

Displays event file log information.

show log file-id [file-id]

Displays event log filter policy information.

show log filter-id [filter-id]

Show log collector statistics for the main, security, change and debug log collectors.

show log log-collector

Displays an event log summary with settings and statistics or the contents of a specific log file, SNMP log, or memory log.

show log log-id [log-id] [severity severity-level] [application application] [sequence from-seq [toseq]] [count number] [subject subject] [ascending | descending]

configure log log-id [log-id] <enter>


23

log-id# info detail

Displays SNMP trap group configuration information.

show log snmp-trap-group [log-id]

Displays syslog event log destination summary information or detailed information on a specific syslog destination.

show log syslog [syslog-id]

3.2. Service mirroring

3.2.1. Service mirroring overview

When troubleshooting complex operational problems, customer packets can be examined as they traverse the network. One way to accomplish this is with an overlay of network analyzers established at multiple PoPs, together with skilled technicians to operate them to decode the data provided. This method of traffic mirroring often requires setting up complex filters in multiple switches and/or routers. These, at best, are only able to mirror from one port to another on the same device. Alcatel’s Service Mirroring extends and integrates these capabilities into the network and provides significant operational benefits. Each 7750 SR can mirror packets from a specific service to any destination point in the network, regardless of interface type or speed. Alcatel’s 7750 SR routers support service-based mirroring. While some Layer 3 switches and routers can mirror on a per-port basis within the device, Alcatel 7750 SR routers can mirror on an n-to-1 unidirectional service basis and re-encapsulate the mirrored data for transport through the core network to another location, using either IP or MPLS tunneling as required Figure 3). Original packets are forwarded while a copy is sent out the mirrored port to the mirroring (destination) port. Service mirroring allows an operator to see the actual traffic on a customer’s service with a ‘sniffer’ sitting in a central location. In many cases, this reduces the need for a separate, costly overlay sniffer network. The mirrored frame size that is to be transmitted to the mirror destination can be explicitly configured by using slicing features. This enables mirroring only the parts needed for analysis. For example, only the headers can be copied for analysis, protecting the integrity and security of customer data, or conversely, copying the full packet, including customer data. Service mirroring is supported on any interface type and on mixed interface types. For example, a service that uses only Ethernet service interfaces can be mirrored to a SONET/SDH network port, transported across the core network and delivered on either Ethernet or SONET/SDH egress ports at the location where service analysis is performed. The packet traffic is uninterrupted and packets flow normally through the mirrored port.


24

Figure 3: Service Mirroring

3.2.2. Mirror implementation

Mirroring can be implemented on ingress or egress service access points (SAPs) or ingress and egress network interfaces. The Flexible Fast Path processing complexes preserve the ingress packet throughout the forwarding and mirroring process, making incremental packet changes on a separate copy. Alcatel’s implementation of packet mirroring is based on two assumptions: • Ingress and egress packets are mirrored as they appear on the wire. This is important for

troubleshooting encapsulation and protocol issues.

o When mirroring at ingress, the Flexible Fast Path network processor array (NPA) sends an exact copy of the original ingress packet to the mirror destination while normal forwarding proceeds on the original packet.

o When mirroring is at egress, the NPA performs normal packet handling on the egress

packet, encapsulating it for the destination interface. A copy of the forwarded packet (as seen on the wire) is forwarded to the mirror destination.

• Mirroring must support tunnel destinations.

o Remote destinations are reached by encapsulating the ingress or egress packet within an SDP, like the traffic for distributed VPN connectivity services. At the remote destination, the tunnel encapsulation is removed and the packet is forwarded out a local SAP.


25

3.2.2.1 Mirror Source and Destinations

Mirror sources and destinations have the following characteristics:

• They can be on the same 7750 SR router (local) or on two different routers (remote). • Mirror destinations can terminate on egress virtual ports which allow multiple mirror

destinations to send to the same packet decode device, delimited by IEEE 802.1Q (referred to as dot1q) tags. This is helpful when troubleshooting a multi-port issue within the network. When multiple mirror destinations terminate on the same egress port, the individual dot1q tags can provide a DTE/DCE separation between the mirror sources.

• Packets ingressing a port can have a mirror destination separate from packets egressing another or the same port (the ports can be on separate nodes).

• A total of 255 mirror destinations are supported (local and/or remote), on a per chassis basis.

The mirror egress port (local or remote) can be PoS or Ethernet. If an Ethernet frame is mirrored to a PoS port, the frame is translated to PPP/BCP encapsulation. If a PoS frame is mirrored to an Ethernet port, the frame is translated to PPPoE encapsulation. This allows the use of PoS or Ethernet packet decode devices.

Local and Remote Mirroring

Mirrored frames can be copied and sent to a specific local destination or service on the 7750 router (local mirroring) or copies can be encapsulated and sent to a different 7750 SR router (remote mirroring). This functionality allows network operators to centralize not only network analyzer (sniffer) resources, but also the technical staff who operate them. The 7750 SR allows multiple concurrent mirroring sessions so traffic from more than one ingress mirror source can be mirrored to the same or different egress mirror destinations. Remote mirroring uses a service distribution path (SDP) which acts as a logical way of directing traffic from one SR-Series router to another through a uni-directional (one-way) service tunnel. The SDP terminates at the far-end 7750 SR which directs packets to the correct destination on that device. The SDP configuration from the mirrored device to a far-end 7750 SR requires a return path SDP from the far-end 7750 SR back to the mirrored router. Each device must have an SDP defined for every remote router to which it wants to provide mirroring services. SDPs must be created first, before services can be configured.


26

Encapsulation Translation

Service mirroring can also map frames from a monitored service to another endpoint using a different encapsulation type at the mirror destination. For example, a service using PPP over Packet over SONET/SDH can have its traffic mirrored to an Ethernet port destination with an Ethernet-attached analyzer. The 7750 SR router translates the PPP header into a PPPoE header so the Ethernet-attached analyzer can properly decode the frames. The automatic translation of PPP or Ethernet frames into PPPoE or BCP encapsulations can be manually disabled. The type of translation depends on the type of the destination SDP or SAP defined for the mirror destination. Translation is important to allow PoS packet-decoding devices to receive Ethernet frames or Ethernet packet-decoding devices to receive PPP frames. When translating an Ethernet frame for transmission to a SONET/SDH SAP or SDP, the Ethernet frame gets encapsulated in a PPP/BCP frame format. When translating a SONET/SDH PPP frame for transmission to an Ethernet SAP or SDP, the PPP frame gets encapsulated in a PPPoE frame format.

Slicing

A further service mirroring refinement is ’slicing’ which copies a specified packet size of each frame. This is useful to monitor network usage without having to copy the actual data. Slicing enables mirroring larger frames than the destination packet decode equipment can handle. It also allows conservation of mirroring resources by limiting the size of the stream of packet through the 7750 SR and the core network. When a mirror slice-size is defined, a threshold that truncates a mirrored frame to a specific size is created. For example, if the value of 256 bytes is defined, up to the first 256 bytes of the frame are transmitted to the mirror destination. The original frame is not affected by the truncation. Mirrored frames, most likely, will grow larger as encapsulations are added when packets are transmitted through the network core or out the mirror destination SAP to the packet/protocol decode equipment. The transmission of a sliced or non-sliced frame is also dependent on the mirror destination SDP path MTU and/or the mirror destination SAP physical MTU. Packets that require a larger MTU than the mirroring destination supports are discarded if the defined slice size does not truncate the packet to an acceptable size.


27

3.2.2.2 Mirroring performance

Replication of mirrored packets can, typically, affect performance and should be used carefully. Alcatel 7750 SR routers minimize the impact of mirroring on performance by taking advantage of its distributed Flexible Fast Path technology. Flexible Fast Path forwarding allows efficient mirror service scaling and, at the same time, allows a large amount of data to be mirrored with minimal performance impact. When a mirror destination is configured, the packet slice option can truncate mirrored packets to the destination, which minimizes replication and tunneling overhead. The mirroring architecture also supports mirror rate limiting both at the ingress and egress Flexible Fast Path NPA. This rate limiting is accomplished through a shaping queue and is settable according to the maximum amount of mirroring desired. Mirroring can be performed based on the following criteria:

• Port • SAP • MAC filter • IP filter • Ingress label

3.2.3. Mirroring configuration

Configuring mirroring is similar to creating a uni-directional service. Mirroring requires the configuration of:

• Mirror source - the traffic on a specific point(s) to mirror. • Mirror destination - the location to send the mirrored traffic, where the sniffer will be

located. Figure 4 depicts a local mirror service configured on SR A.

• Port 2/1/2 is specified as the source. Mirrored traffic ingressing and egressing this port will be sent to port 2/1/3.

• SAP 2/1/3 is specified as the destination. The sniffer is physically connected to this port. Mirrored traffic ingressing and egressing port 2/1/2 is sent here. SAP, encapsulation requirements, packet slicing, and mirror classification parameters are configured. SDPs are not used in local mirroring.


28

Figure 4: Local mirroring Example

Figure 5 depicts a remote mirror service configured as SR B as the mirror source and SR A as the mirror destination. Mirrored traffic ingressing and egressing port 5/2/1 (the source) on SR B is handled the following ways:

• Port 5/2/1 is specified as the mirror source port. Parameters are defined to select specific traffic ingressing and egressing this port.

• Destination parameters are defined to specify where the mirrored traffic will be sent. In this

case, mirrored traffic will be sent to a SAP configured as part of the mirror service on port 3/1/3 on SR A (the mirror destination).

• SR A decodes the service ID and sends the traffic out of port 3/1/3.

• The sniffer is physically connected to this port (3/1/3). SAP, encapsulation requirements,

packet slicing, and mirror classification parameters are configured in the destination parameters.


29

Figure 5: Remote mirroring Example

3.2.3.1 Mirror configuration process overview Figure 6 displays the process to provision basic mirroring parameters.

Figure 6: Service mirror configuration and implementation flow

3.2.3.2 Mirror configuration components

The example below demonstrates the major components to configure service mirroring.


30

• Mirror destination — Sets up a service which allows the mirrored packets to be directed locally or over the core of the network and have a far end 7750 SR decode the mirror encapsulation. The service ID must match in the mirror-destination and the mirror-source context. • SAP (mirror destination) — Creates a service access point (SAP), which defines the port and encapsulation parameters to which the mirrored source packets are sent. The sniffer is physically connected to this port. • SDP — For remote mirrored service. Binds an existing (mirror) service distribution path (SDP) to the mirror destination service ID to transport the source mirrored traffic to the destination. • Remote source — For remote mirrored services. Specifies the remote (source) SR allowed to mirror traffic to this device for mirror service egress. • Mirror source — Configures packet mirroring match criteria for a mirror destination service. The same mirror destination service ID and the mirror source service ID must be configured. • Port — A packet mirroring option which defines ingress and/or egress traffic monitoring by port. • SAP (mirror source) — A packet mirroring option which defines ingress and/or egress traffic monitoring by SAP defined by the port-id:encap-val or portid.channel-id:encap-val. • IP filter — A packet mirroring option which specifies that packets matching the IP filter are mirrored to a mirror destination. • MAC filter — A packet mirroring option which specifies that packets matching the MAC filter are mirrored to a mirror destination. • Ingress label — A packet mirroring option which defines packets with a specific MPLS label to a mirror destination.

3.2.3.3 Basic mirror configuration Example

Local Service mirroring configuration Each local mirrored service (within the same router) requires the following configurations: 1. Specify mirror destination (SAP, SDP). 2. Specify mirror source (port, SAP, SDP, IP filter, MAC filter, ingress label). Note that the mirror source and mirror destination components must be configured under the same service ID context.


31

Figure 7: Local Service Mirroring Configuration

The following example displays a sample configuration for Figure 7 of a local mirrored service where the source and destinations are on the same SR (SR1). SRA>config>mirror# info ----------------------------------------------

mirror-dest 103 create sap 2/1/3:0 create

egress qos 1

exit exit no shutdown

exit ---------------------------------------------- SRA>config>mirror#

The following displays the mirror source configuration: SRA>debug>mirror-source# show debug mirror debug

mirror-source 103 port 2/1/2 egress ingress no shutdown

exit exit SR1>debug>mirror-source# exit Remote Service mirroring configuration

Each remote mirrored service (across the network core) requires the following configurations: 1. Define the remote destination (SDP) 2. Identify the remote source (the device allowed to mirror traffic to this device)


32

3. Specify the mirror destination (SAP) 4. Specify mirror source (port, SAP, SDP, IP filter, MAC filter, ingress label) Note that the mirror source and mirror destination components must be configured under the same service ID context.

Figure 8: Remote Service Mirroring Configuration

The following example displays a sample configuration of a remote mirrored service for Figure 8 where the source is a port on SRB and the destination is a SAP on SRA. SRB>config>mirror# info ----------------------------------------------

mirror-dest 1000 create sdp 2 egr-svc-label 7000 no shutdown

exit ---------------------------------------------- SRB>config>mirror# exit all SRB# show debug debug

mirror-source 1000 port 5/2/1 egress ingress no shutdown

exit exit SRB# SRA>config>mirror# info ----------------------------------------------

mirror-dest 1000 create remote-source

far-end 10.10.10.104 ing-svc-label 7000 exit sap 3/1/3:0 create

egress qos 1

exit exit no shutdown

exit ---------------------------------------------- SRA>config>mirror#


33

3.2.3.4 Mirror configuration Notes

This section describes limitations or notes regarding mirroring configuration.

• Up to 255 mirroring service IDs may be created within a single system. • A mirrored source can only have one destination. • The destination mirroring service IDs and service parameters are persistent between router

(re)boots and are included in the configuration saves.

The source packet mirroring enabling criteria defined in debug mirror mirror-source commands are not preserved in configuration saves.

• Physical layer problems such as collisions, jabbers, etc., are not mirrored. Typically, only complete packets are mirrored. An exception to this is that packets with CRC errors are mirrored. Complete stats are available on the interface for these physical layer problems.

• SONET ports or channels in access mode and with frame-relay encapsulation types cannot be mirrored.

• Either LAG ports or LAG port members can be mirrored. If a LAG port member is being mirrored, then the LAG port cannot be mirrored and vice-versa.

• Clear channel ports (TDM or SONET) that are being mirrored cannot be channelized until the mirroring is disabled.

• Encap type on an access port/channel can not be changed to frame-relay if it is being mirrored.

• Starting and shutting down mirroring: Mirror destinations:

• The default state for a mirror destination service ID is shutdown. You must issue a no shutdown command to enable the feature.

• When a mirror destination service ID is shutdown, mirrored packets associated with the service ID are not accepted from its mirror source or remote source 7750 SR router. The associated mirror source is put into an operationally down mode. Mirrored packets are not transmitted out the SAP or SDP. Each mirrored packet is silently discarded. If the mirror destination is a SAP, the SAP’s discard counters are incremented.

• Issuing the shutdown command causes the mirror destination service or its mirror source to be put into an administratively down state. Mirror destination service IDsmust be shut down first in order to delete a service ID, SAP, or SDP association from the system.

Mirror sources:

• The default state for a mirror source for a given mirror-dest service ID is no shutdown. You must enter a shutdown command to deactivate (disable) mirroring from that mirror-source.

• Mirror sources do not need to be shutdown to remove them from the system. When a mirror source is shutdown, mirroring is terminated for all sources defined locally for the mirror destination service ID.


34

3.2.3.5 List of CLI commands to configure Mirroring parameters

Table 4 lists all the configuration commands to configure 7750 SR mirroring parameters, indicating the configuration level at which each command is implemented with a short command description. The command list is organized in the following task-oriented manner:

• Configure mirror destination parameters • Configure mirror source parameters • Configure an SDP


35

Table 4: CLI Commands to Configure Mirroring Parameters


36

Show command show mirror mirror-dest [service- id] Displays mirror configuration and operation

information.

3.3. OA&M commands for troubleshooting

Proper delivery of services requires a number of operations occur properly and at different levels in the service delivery model. For example, operations such as the association of packets to a service, VC-labels to a service and each service to a service tunnel must be performed properly in the forwarding plane for the service to function properly. In order to verify that a service is operational, a set of in-band, packet-based OAM tools is required, with the ability to test each of the individual packet operations. For in-band testing, the OAM packets closely resemble customer packets to effectively test the customer’s forwarding path, but they are distinguishable from customer packets so they are kept within the service provider’s network and not forwarded to the customer. The 7750 SR OS suite of OAM diagnostics supplement the basic IP ping and traceroute operations with diagnostics specialized for the different levels in the service delivery model. There are diagnostics for MPLS LSPs, SDPs, Services and VPLS MACs within a service.

3.3.1. LSP Diagnostics

The 7750 SR OS LSP diagnostics are implementations of LSP ping and LSP traceroute based on Internet Draft draft-ietf-mpls-lsp-ping-02.txt. LSP ping, as described in the draft, provides a mechanism to detect data plane failures in MPLS LSPs. LSP ping and LSP traceroute are modeled after the ICMP echo request/reply used by ping and traceroute to detect and localize faults in IP networks. For a given FEC, LSP ping verifies whether the packet reaches the egress label edge router (LER), while in LSP traceroute mode, the packet is sent to the control plane of each transit label switched router (LSR) which performs various checks to see if it is actually a transit LSR for the path.

3.3.2. SDP Diagnostics

The 7750 SR OS SDP diagnostics are SDP Ping and SDP MTU Path Discovery. SDP Ping

SDP Ping performs in-band uni-directional or round-trip connectivity tests on SDPs. The SDP Ping OAM packets are sent in-band, in the tunnel encapsulation, so it will follow the same path as traffic within the service. The SDP Ping response can be received out-of-band in the control plane, or in-band using the data plane for a round-trip test. For a unidirectional test, SDP Ping tests:


37

• Egress SDP ID encapsulation • Ability to reach the far-end IP address of the SDP ID within the SDP encapsulation • Path MTU to the far-end IP address over the SDP ID • Forwarding class mapping between the near-end SDP ID encapsulation and the far-end tunnel termination

For a round-trip test, SDP Ping uses a local egress SDP ID and an expected remote SDP ID. Since SDPs are unidirectional tunnels, the remote SDP ID must be specified and must exist as a configured SDP ID on the far-end 7750 SR. SDP round trip testing is an extension of SDP connectivity testing with the additional ability to test:

• Remote SDP ID encapsulation • Potential service round trip time • Round trip path MTU • Round trip forwarding class mapping

SDP MTU Path Discovery

In a large network, network devices can support a variety of packet sizes that are transmitted across its interfaces. This capability is referred to as the Maximum Transmission Unit (MTU) of network interfaces. It is important to understand the MTU of the entire path end-to-end when provisioning services, especially for virtual leased line (VLL) services where the service must support the ability to transmit the largest customer packet. The Path MTU Discovery tool provides a powerful tool that enables service provider to get the exact MTU supported between the service ingress and service termination points (accurate to one byte).

3.3.3. Service Diagnostics

Alcatel’s Service Ping feature provides end-to-end connectivity testing for an individual service. Service Ping operates at a higher level than the SDP diagnostics in that it verifies an individual service and not the collection of services carried within an SDP. Service Ping is initiated from a 7750 SR router to verify round-trip connectivity and delay to the far-end of the service. Alcatel’s implementation functions for both GRE and MPLS tunnels and tests the following from edge-to-edge:

• Tunnel connectivity • VC label mapping verification • Service existence • Service provisioned parameter verification • Round trip path verification • Service dynamic configuration verification


38

3.3.4. VPLS MAC Diagnostics

While the LSP ping, SDP ping and Service ping tools enable transport tunnel testing and verify whether the correct transport tunnel is used, they do not provide the means to test the learning and forwarding functions on a per-VPLS-service basis. It is conceivable, that while tunnels are operational and correctly bound to a service, an incorrect Forwarding Information Base (FIB) table for a service could cause connectivity issues in the service and not be detected by the ping tools. Alcatel has developed VPLS OAM functionality to specifically test all the critical functions on a per-service basis. These tools are based primarily on the IETF document draft-stokes-vkompella-ppvpn-hvpls-oam-00.txt. The 7750 SR VPLS OAM tools include: • MAC Ping — Provides the ability to trace end-to-end switching of specified MAC addresses. MAC ping provides an end-to-end test to identify the egress customer-facing port where a customer MAC was learned. MAC ping can also be used with a broadcast MAC address to identify all egress points of a service for the specified broadcast MAC. • MAC Trace — Provides the ability to trace a specified MAC address hop-by-hop until the last node in the service domain. • MAC Populate — Allows specified MAC addresses to be injected in the VPLS service domain. This triggers learning of the injected MAC address by all participating nodes in the service. This tool is generally followed by MAC ping or MAC trace to verify if correct learning occurred. • MAC Purge — Allows MAC addresses to be flushed from all nodes in a service domain. MAC Ping

For a MAC ping test, the destination MAC address (unicast or multicast) to be tested must be specified. A MAC ping packet can be sent through the control plane or the data plane. When sent by the control plane, the ping packet goes directly to the destination IP in a UDP/IP OAM packet. If it is sent by the data plane, the ping packet goes out with the data plane format. In the control plane, a MAC ping is forwarded along the flooding domain if no MAC address bindings exist. If MAC address bindings exist, then the packet is forwarded along those paths (if they are active). Finally, a response is generated only when there is an egress SAP binding to that MAC address. A control plane request is responded to via a control reply only. In the data plane, a MAC ping is sent with a VC label TTL of 255. This packet traverses each hop using forwarding plane information for next hop, VC label, etc. The VC label is swapped at each service-aware hop, and the VC TTL is decremented. If the VC TTL is decremented to 0, the packet is passed up to the management plane for processing. If the packet reaches an egress node, and would be forwarded out a customer facing port, it is identified by the OAM label below the VC label and passed to the management plane. MAC pings are flooded when they are unknown at an intermediate node. They are responded to only by the egress nodes that have mappings for that MAC address.


39

MAC Trace

A MAC trace functions like an LSP trace with some variations. Operations in a MAC trace are triggered when the VC TTL is decremented to 0. Like a MAC ping, a MAC trace can be sent either by the control plane or the data plane. For MAC trace requests sent by the control plane, the destination IP address is determined from the control plane mapping for the destination MAC. If the destination MAC is known to be at a specific remote site, then the far-end IP address of that SDP is used. If the destination MAC is not known, then the packet is sent unicast, to all SDPs in the service with the appropriate squelching. A control plane MAC traceroute request is sent via UDP/IP. The destination UDP port is the LSP ping port. The source UDP port is whatever the system gives (note that this source UDP port is really the demultiplexor that identifies the particular instance that sent the request, when correlating the reply). The source IP address is the system IP of the sender. When a traceroute request is sent via the data plane, the data plane format is used. The reply can be via the data plane or the control plane. A data plane MAC traceroute request includes the tunnel encapsulation, the VC label, and the OAM, followed by an Ethernet DLC, a UDP and IP header. If the mapping for the MAC address is known at the sender, then the data plane request is sent down the known SDP with the appropriate tunnel encapsulation and VC label. If it is not known, then it is sent down every SDP (with the appropriate tunnel encapsulation per SDP and appropriate egress VC label per SDP binding). The tunnel encapsulation TTL is set to 255. The VC label TTL is initially set to the min-ttl (default is 1). The OAM label TTL is set to 2. The destination IP address is the all-routers multicast address. The source IP address is the system IP of the sender. The destination UDP port is the LSP ping port. The source UDP port is whatever the system gives (note that this source UDP port is really the demultiplexor that identifies the particular instance that sent the request, when correlating the reply). The Reply Mode is either 3 (i.e., reply via the control plane) or 4 (i.e., reply via the data plane), depending on the reply-control option. By default, the data plane request is sent with Reply Mode 3 (control plane reply). The Ethernet DLC header source MAC address is set to either the system MAC address (if no source MAC is specified) or to the specified source MAC. The destination MAC address is set to the specified destination MAC. The ethertype is set to IP. MAC Populate

MAC Populate is used to send a message through the flooding domain to learn a MAC address as if a customer packet with that source MAC address had flooded the domain from that ingress point in the service. This allows the provider to craft a learning history and engineer packets in a particular way to test forwarding plane correctness.


40

The MAC populate request is sent with a VC TTL of 1, which means that it is received at the forwarding plane at the first hop and passed directly up to the management plane. The packet is then responded to by populating the MAC address in the forwarding plane, like a conventional learn although the MAC will be an OAM-type MAC in the FIB to distinguish it from customer MACs addresses. This packet is then taken by the control plane and flooded out the flooding domain (squelching appropriately, the sender and other paths that would be squelched in a typical flood). This controlled population of the FIB is very important to manage the expected results of an OAM test. The same functions are available by sending the OAM packet as a UDP/IP OAM packet. It is then forwarded to each hop and the management plane has to do the flooding. Options for MAC Populate are to force the MAC in the table to type OAM (in case it already existed as dynamic or static or an OAM induced learning with some other binding), to prevent new dynamic learning to over-write the existing OAM MAC entry, to allow customer packets with this MAC to either ingress or egress the network, while still using the OAM MAC entry. Finally, an option to flood the MAC Populate request causes each upstream node to learn the MAC (i.e., populate the local FIB with an OAM MAC entry), and to flood the request along the data plane using the flooding domain. An age can be provided to age a particular OAM MAC after a different interval than other MACs in a FIB. MAC Purge

MAC Purge is used to clear the FIBs of any learned information for a particular MAC address. This allows one to do a controlled OAM test without learning induced by customer packets. In addition to clearing the FIB of a particular MAC address, the purge can also indicate to the control plane not to allow further learning from customer packets. This allows the FIB to be clean, and be populated only via a MAC Populate. MAC Purge follows the same flooding mechanism as the MAC Populate. A UDP/IP version of this command is also available that does not follow the forwarding notion of the flooding domain, but the control plane notion of it.

3.3.5. OAM Command Summary

LSP diagnostic commands

oam lsp-ping In-band LSP ping utility to verify LSP connectivity

oam lsp-trace In-band LSP traceroute command to determine the hop-by-hop path for an LSP.


41

SDP diagnostic commands

oam sdp-mtu Performs in-band MTU Path tests on an SDP to determine the largest path-mtu supported on an SDP.

oam sdp-ping Tests an SDP for in-band uni-directional or round trip connectivity with a round trip time estimate.

Service diagnostic commands

oam svc-ping Tests a service ID for correct and consistent provisioning between two service end points. The following information can be determined from svc-ping: • Local and remote service existence • Local and remote service state • Local and remote service type correlation • Local and remote customer association • Local and remote service-to-SDP bindings and state • Local and remote ingress and egress service label association

VPLS MAC diagnostic commands

oam mac-ping In-band and out-of-band utility to determine the existence of an egress SAP binding of a given MAC within a VPLS. Utility can also be used to display all operationally up SAPs in the VPLS service.

oam mac-populate Populates the FIB with an OAM-type MAC entry indicating the node is the egress node for the MAC address and optionally floods the OAM MAC association throughout the service

oam mac-purge Removes an OAM-type MAC entry from the FIB and optionally floods the OAM MAC removal throughout the service.

oam mac-trace In-band or out-of-band utility to determine the hop-by-hop path for a destination MAC address within a VPLS.


42

4. Hardware Operational Status

Verifying the operational status of the hardware is similar to what a mechanic will do to troubleshoot a problem you have reported on your car. Cars can be broken down into mechanical sub-components, such as the body, the suspension, the engine, the transmission, the electrical, that help in troubleshooting by focusing the root cause analysis on a particular component that is related to the problem symptom. For example, if the service engine trouble light comes on, the mechanic is not likely to start examining the suspension but rather will focus his attention on the engine.

Similarly, the Alcatel 7750 SR can be broken down into hardware sub-components that have built-in mechanisms to report problems at the hardware and hardware configuration levels. These are described in the following section.

4.1. 7750 SR-12 Hardware Overview

In the 7750 SR-12 chassis, the input/output module (IOM) slots are numbered 1 through 10. The card slots are vertically oriented. A maximum of two MDAs can be installed on each IOM. MDAs are installed in either MDA slot 1 (top slot) or MDA slot 2 (bottom slot) on an IOM.

A maximum of two SF/CPMs can be installed in the center SF/CPM slots which are designated as slots A and B. At least one SF/CPM must be installed in order for the router to operate. The redundant SF/CPM operates in standby mode and takes over system operation if the primary fails.

The 7750 SR-12 provides access to components from both the front and back sides. The filter tray, SF/CPMs, IOMs, and MDAs are accessed from the front of the chassis. The power entry modules (PEMs) and cooling trays (impeller trays) are accessible from the chassis rear. Figure 9 and Figure 10 show front and rear views.

DC PEMs are horizontally oriented and are accessed through the lower rear of the chassis. The slots are designated as “1” for the top slot and “2” for the lower slot. The DC PEMs can be connected directly to a DC power source. Optionally, power can be obtained through AC power rectifiers.

The mounting brackets for the chassis are factory installed to mount in a standard 19-inch wide rack.


43

Figure 9: 7750 SR-12 Chassis Front View

1 Cable management system

2 Chassis slot numbers

3 MDA (installed)

4 Full slot panel blank

5 SF/CPM

6 MDA blank panel

7 Rack mounting brackets

8 Air vent

9 ESD plug

10 Compact flash slots

11 Compact flash slot 3 (cf3:)

Table 5: Chassis Front View Features


44

Figure 10: 7750 SR-12 Chassis Rear View

1 Grounding studs

2 Rack mounting brackets

3 Impeller (fan) trays

4 VDC studs for DC power cable

5 RTN studs for DC power cable

6 Safety cover

7 OFF/ON DC switch

8 Impeller (fan) tray faceplate

9 DC PEMs. The top slot is referred to as PEM Slot 1. The lower slot is referred to as PEM Slot 2.

10 DB-25 connector (status)

Table 6: Chassis Rear View Features


45

4.2. Verifying Router Boot Sequence

The compact flash card must be installed in compact flash card slot #3 (cf3) in order for the router to initialize.

If the system cannot load or cannot find the boot.ldr file on cf3, the system checks for a manual boot sequence interruption. Unless an unsuccessful system initialization is manually interrupted, the system will continuously reboot in an attempt to successfully find and load the boot.ldr file. Load a compact flash card with the appropriate boot.ldr file into the cf3 slot. When the system finds the boot.ldr file, the system processes the initialization parameters from the BOF. The BOF should be on the same drive as the boot loader file. If the BOF cannot be found or loaded, then the system prompts for a different image and configuration location. When the image is successfully loaded, control is passed from the boot loader file to the image. The runtime image attempts to locate the configuration file as configured in the BOF. The configuration file includes chassis, IOM, MDA, and port configurations, as well as system, routing, and service configurations.

The show boot-messages command can be used to display boot messages from the last system restart. An example of the output of this command can be seen in the reference 7750_SR_OS_System_Guide_2.0.pdf

4.3. Verifying Management Connection Operational Status

4.3.1. Console Port Management Connection

Management access to the 7750 SR is supported through a local console connection, illustrated in Figure 11.

Figure 11: Management Console Port Connection

Troubleshooting a Console Connection

If you are unable to bring up a management session through the console port connection, the most likely source of the problem is the console configuration. It should be configured as in Table 7 below.


46

Table 7: Console Configuration Parameter Values

You should also verify the DTE/DCE setting of the terminal and select the appropriate setting for the console port. The pinout assignment for the console port connector for both DTE and DCE settings is available in the 7750_SR-12_Installation_Guide_Rev-02.

4.3.2. Telnet Management Connection

Management access to the 7750 SR is also supported through a telnet connection to the management port, illustrated in Figure 12.

Figure 12: Telnet Management Port Connection

Troubleshooting a Telnet Connection

If you are unable to bring up a management session through the console port connection, verify that the management port has been assigned an IP address by issuing a show bof command from a management session established through the console port or an IP interface on the router.

4.4. Verifying Chassis Operational Status

4.4.1. Chassis Configurations

Table 8 below lists the operating requirements of the various hardware components of the 7750 SR-12 that makeup a minimum and a maximum chassis configuration. The chassis must contain at least one SF/CPM, at least one flash memory card installed in Slot #3 in SF/CPM, at least one IOM and at least one MDA.


47

Table 8: 7750 SR-12 Hardware Component Operating Requirements

The 7750 SR-12 is equipped with critical, major and minor alarm LEDs that provide a visual indication that a critical, major or minor alarm exists somewhere in the router, be that with either with the hardware, hardware configuration, router sub-systems, routing or service environment. The show chassis command can be used to display any current error conditions that may exist in the router. The following is an example of the output for this command:


48

4.4.2. Things to Check - Power Supply

Figure 13 illustrates the power supply LEDs and Table 9 provides the LED descriptions. If a fault condition exists verify that the power is connected, voltage is present and the chassis ground connection is sound. Check the cooling system and air filter condition and service if required. If the fault condition persists, change the power supply.

Figure 13: 7750 SR-12 AC Power Supply LEDs

1 AC OK Green: the unit has input AC in the correct range

2 DC OK Green: the unit is powered up and the output is in regulation

3 Fault Red: The unit has detected an internal fault

Table 9: 7750 SR-12 AC Power Supply LED Descriptions


49

The show chassis and show chassis power-supply commands will display the current status of the router power supply indicating any error conditions. The following is an example of the output of these commands:

4.4.3. Things to Check - Fans

The show chassis and show chassis environment commands will display the current status of the router fans indicating any error conditions. The following is an example of the output of these commands:


50

4.5. Verifying SF/CPM Operational Status

4.5.1. Minimum Configuration

• At least one SF/CPM must be installed

• At least one flash memory card be installed in Slot #3 in SF/CPM

4.5.2. SF/CPM LED Status

Verify proper operational status by checking the Power and Status LEDs on the active CPM faceplate as illustrated in Figure 14. Table 10 provides the field descriptions and indications of potential problem conditions. For more detail refer to 7750_SR-12_Installation_Guide_Rev-02.

Figure 14: SF/CPM Front Panel

Key Indicator Category Potential Problem Indication

3 Status Amber: Operationally down but administratively up.

Unlit: Not operational, shutdown, or administratively down.

3 M/S (Master/Slave)

Ctl Green (blinking): Indicates that the SF/CPM is operating as the secondary SF/CPM in a redundant configuration.

3 M/S (Master/Slave)

Ref Green (blinking): Indicates that the SF/CPM is operating as the secondary clocking reference in a redundant system.

Unlit: Clock not initialized.

3 Timing Green (blinking): Clock in (internal) holdover state

Amber (blinking): Clock in free running state

Unlit: Clock not initialized.


51

3 Reference 1,2 Amber: The reference is enabled (no shutdown) but not qualified.

Unlit: Not in use, not configured.

3 Reference 3 BITS Status:

Amber: The reference is enabled (no shutdown) but not qualified.

Unlit: Not in use, not configured.

3 Power Supply 1,2,3,4 Amber: Indicates an error condition with an installed power entry module in the associated slot.

Unlit: Indicates that a power entry module is not installed or not recognized.

3 Fan Status 1,2,3 Amber: Indicates a fan tray failure.

Unlit: Indicates that a fan tray is not installed.

3 Compact Flash 1,2,3 Amber (blinking): Error condition exists.

Amber (solid): Indicates that the slot is in an operationally down mode. (This is the only mode to safely remove the flash card.)

Unlit: A flash card is not installed in the slot.

3 Alarms OT Red: An over-temperature condition exists.

3 Alarms Crit Red: A critical condition exists, such as a severe over-temperature condition, a fan tray failure, an over-current condition in a power module, or an out-of-tolerance voltage.

3 Alarms Maj Red: A serious condition exists, such as an over-temperature condition, a fan tray failure, an over-current condition in a power module, or an out-of-tolerance voltage.

3 Alarms Min Amber: A serious condition exists, such as a component failure.

10 Mgmt Link Unlit: Operationally down.

10 Mgmt Data Amber (blinking): Error condition.

Table 10: SF/CPM Field Descriptions

4.5.3. CLI commands for SF/CPM troubleshooting

Below are some CLI commands used for troubleshooting an issue related to SF/CPM:


52

Task Recommended CLI command(s)

1 To display the SF/CPM card status show card

2 To switchover to standby SF/CPM card (assuming the standby card is up)

admin reboot active [now]

3 To verify the switchover show card

Examples of command output:

1. show card

SR12# show card =============================================================================== Card Summary =============================================================================== slot card card card admin operational allowed provisioned equipped state state ------------------------------------------------------------------------------- 1 all supported iom-20g iom-20g up up 2 all supported iom-20g up down 3 all supported iom-20g up down 6 all supported iom-20g up down 9 all supported iom-20g up down A all supported sfm-400g sfm-400g up up/active B all supported sfm-400g sfm-400g up up/standby ===============================================================================

2. admin reboot active [now]

(Before the switchover) SR12# show card =============================================================================== Card Summary =============================================================================== slot card card card admin operational allowed provisioned equipped state state ------------------------------------------------------------------------------- 1 all supported iom-20g iom-20g up up 2 all supported iom-20g up down 3 all supported iom-20g up down 6 all supported iom-20g up down 9 all supported iom-20g up down A all supported sfm-400g sfm-400g up up/active B all supported sfm-400g sfm-400g up up/standby =============================================================================== SR12# admin reboot active now


53

(After the switchover) SR12# show card =============================================================================== Card Summary =============================================================================== slot card card card admin operational allowed provisioned equipped state state ------------------------------------------------------------------------------- 1 all supported iom-20g iom-20g up up 2 all supported iom-20g up down 3 all supported iom-20g up down 6 all supported iom-20g up down 9 all supported iom-20g up down A all supported sfm-400g sfm-400g up up/standby B all supported sfm-400g sfm-400g up up/active ===============================================================================

4.5.4. CLI commands for SF/CPM health check

Below are some CLI commands used to check SF/CPM health in several aspects. More commands can be found in Section 5.3.

Task Recommended CLI command(s)

1 To check the status of the SF/CPM card

(Note: the <slot-number> of SF/CPM on slot A is “A”, on slot B is “B”.)

show card <slot-number> detail

2 To check if there is any alarm/log related to the SF/CPM card

show log log-id <log-id> subject <subject>

(Notes:

The <subject> here is “Card A” if it is to check the SF/CPM on Slot A; or “Card B” if it is to check the SF/CPM on Slot B..

The subject string is Case Sensitive.)

3 To display system cpu

show system cpu

4 To display system memory

show system memory-pools

5 To display system uptime show system info

Examples of command output:

1. show card <slot-number> detail


54

SR12# show card A detail =============================================================================== Card A =============================================================================== slot card card card admin operational allowed provisioned equipped state state ------------------------------------------------------------------------------- A sfm-400g sfm-400g sfm-400g up up/active sfm-200g BOF last modified : N/A Config file version : Config file last modified : N/A Config file last saved : N/A CPM card status : active Flash - cf1: Administrative State : up Operational state : not equipped Flash - cf2: Administrative State : up Operational state : not equipped Flash - cf3: Administrative State : up Operational state : up Serial number : 103616B2304W340 Firmware revision : HDX 2.1 Model number : SanDisk SDCFB-128 Size : 125,038 KB Free space : 96,836 KB Hardware Data Part number : 3HE00018AAAA01 CLEI code : Serial number : NS041410366 Manufacture date : 04112004 Manufacturing string : Manufacturing deviations : Administrative state : up Operational state : up Status : software running Temperature : 44C Temperature threshold : 68C Software boot version : X-2.0.R1 on Tue May 4 15:07:26 PST 2004 by* Software version : TiMOS-C-2.0.R4 cpm/hops ALCATEL SR 7750 Co* Time of last boot : 2004/09/07 08:16:04 Current alarm state : alarm cleared Base MAC address : 00:03:fa:0c:e4:4a Memory capacity : 2,016 MB ===============================================================================

2. show log log-id <log-id> subject <subject>


55

SR12>show>log# log-id 99 subject "Card A" =============================================================================== Event Log 99 =============================================================================== Description : Default System Log Memory Log contents [size=500 next event=20 (not wrapped)] 6 2004/07/19 06:37:41.48 MINOR: CHASSIS #2002 - Card A "Class CPM Module : inserted"

3. show system cpu

SR12# show system cpu ========================================= CPU Utilization (Test time 1001407 uSec) ========================================= Name CPU Time CPU Usage (uSec) ----------------------------------------- System 1427 0.14% Icc 50 ~0.00% RTM/Policies 0 0.00% OSPF 0 0.00% MPLS/RSVP 0 0.00% LDP 0 0.00% IS-IS 0 0.00% RIP 0 0.00% VRRP 0 0.00% BGP 0 0.00% Services 4 ~0.00% IOM 5607 0.55% SIM 79 ~0.00% CFLOWD 0 0.00% Idle 994240 99.28% =========================================

4. show system memory-pools SR12# show system memory-pools =============================================================================== Memory Pools =============================================================================== Name Max Allowed Current Size Max So Far In Use ------------------------------------------------------------------------------- System No limit 118,489,688 118,489,688 114,333,488 Icc 8,388,608 1,048,576 1,048,576 33,616 RTM/Policies No limit 4,194,336 4,194,336 2,507,136 OSPF No limit 0 0 0 MPLS/RSVP No limit 1,048,576 1,048,576 76,000 LDP No limit 0 0 0 IS-IS No limit 0 0 0 RIP No limit 0 0 0


56

VRRP No limit 0 0 0 BGP No limit 0 0 0 Services No limit 2,097,152 2,097,152 1,700,136 IOM No limit 199,156,416 199,156,416 195,826,168 SIM No limit 1,048,576 1,048,576 392 CFLOWD No limit 0 1,048,576 0 ------------------------------------------------------------------------------- Current Total Size : 327,083,320 bytes Total In Use : 314,476,936 bytes Available Memory : 640,711,688 bytes ===============================================================================

5. show system info SR12# show system information ====================================================================== System Information ====================================================================== System Name : sim9 System Contact : System Location : System Coordinates : System Up Time : 3 days, 20:20:40.40 (hr:min:sec) SNMP Port : 161 SNMP Engine ID : 0000197f000000008eb1ff00 SNMP Max Message Size : 1500 SNMP Admin State : Disabled SNMP Oper State : Disabled SNMP Index Boot Status : Not Persistent BOF Source : cf1: Image Source : primary Config Source : N/A Last Booted Config File: N/A Last Boot Cfg Version : N/A Last Boot Config Header: N/A Last Boot Index Version: N/A Last Boot Index Header : N/A Last Saved Config : N/A Time Last Saved : N/A Changes Since Last Save: No Max Cfg/BOF Backup Rev : 5 Cfg-OK Script : N/A Cfg-OK Script Status : not used Cfg-Fail Script : N/A Cfg-Fail Script Status : not used Management IP Addr : 138.120.199.177/24 DNS Server : 138.120.118.196 DNS Domain : ca.newbridge.com BOF Static Routes : To Next Hop 138.120.0.0/16 138.120.199.1 128.251.10.0/24 138.120.199.1 ======================================================================


57

4.6. Verifying IOM Operational Status

On a 7750 SR-12, line cards (IOMs) are only designed to be installed in slots 1 through 10, that is the five left-most and five right-most card slots. The middle two slots are for the SF/CPM cards. Chassis slots must be pre-provisioned to accept specific IOM types. IOMs installed in an un-provisioned chassis slot will remain administratively and operationally down.

When an IOM is installed in a slot and enabled, the system verifies that the installed IOM type matches the allowed IOM type. The IOM will remain offline if the parameters do not match. To see the IOM configuration at system initialization use the show boot-messages command. To display the current IOM configuration use the show card command. The following is an example of the output for this command:

To reset an IOM as part of troubleshooting IOM, use the command: clear card <slot-number>. This command reinitializes the card in the specified slot.

The following is an example of the result of reset an IOM.

SR12# clear card 1/2/3 SR12# show log log-id 99 subject "Card 1" =============================================================================== Event Log 99 =============================================================================== Description : Default System Log Memory Log contents [size=500 next event=292 (not wrapped)] 291 2004/07/28 14:28:35.57 MINOR: CHASSIS #2002 - Card 1 "Class IO Module : inserted" 288 2004/07/28 14:28:16.77 MINOR: CHASSIS #2003 - Card 1 "Class IO Module : removed"

To display the last time IOM was reset, use the show card <slot-number> detail command. The following is an example of the output for this command:

SR12# show card 1 detail


58

=============================================================================== Card 1 =============================================================================== slot card card card admin operational allowed provisioned equipped state state ------------------------------------------------------------------------------- 1 iom-10g iom-20g iom-20g up up iom-20g IOM Card Specific Data Clock source : none Available MDA slots : 2 Installed MDAs : 2 Hardware Data Part number : 3HE00020AAAA01 CLEI code : Serial number : NS041110257 Manufacture date : 03192004 Manufacturing string : Manufacturing deviations : Administrative state : up Operational state : up Status : software running Temperature : 56C Temperature threshold : 68C Software boot version : X-2.0.R1 on Tue May 4 15:07:26 PST 2004 by* Software version : TiMOS-I-2.0.R5 iom/hops ALCATEL SR 7750 Co* Time of last boot : 2004/07/28 14:29:11 Current alarm state : alarm cleared Base MAC address : 00:03:fa:0c:e6:88 ===============================================================================

4.7. Verifying MDA Operational Status

IOMs must be provisioned to accept specific MDA types. MDAs installed in an un-provisioned IOM or chassis card slot will remain administratively and operationally down.

Once an MDA is installed and enabled, the system verifies that the installed type matches the allowed MDA type. The MDA remains offline if the parameters do not match. To display the current MDA configuration use the show mda <slot-id> command. The following is an example of the output for this command:


59

The show mda <slot-id> detail will display any alarm conditions that exist for that MDA. The following information on MDA error conditions can be obtained using this command:


60

5. System level configuration verification

This section provides information about verifying system level configurations of a 7750 SR. It covers the hardware initialization, BOF configuration, CPM redundancy configuration, timing configuration, Security access configuration, and card/MDA/port configurations. Commonly used CLI commands for troubleshooting are provided.

5.1. Summary of system configuration verification

Table 11 provides an index of tasks to verify system level configurations. Detailed description of each task and corresponding CLI commands or notes can be found in the sections indicated in the table.

Table 11: Index of system configuration verification tasks

System Area Tasks Section

System initialization

bof.cfg file not found 5.2

Display current system configuration

Display the BOF configuration

Modify a BOF configuration

Save a BOF configuration

Reboot

5.2.1 BOF configuration

Troubleshooting notes on BOF configuration 5.2.2

Display system information 5.3.1

Display SF/CPMs redundancy configuration

Automatically synchronize two SF/CPMs

Manually synchronize two SF/CPMs

5.3.2

Timing configuration

Change a timing reference input mode to be revertive or non-revertive

Force the system timing output to use a specific reference

5.3.3

System management configuration

SNTP configuration 5.3.4

Display Authentication configuration


61

Display Authentication configuration

Display Authorization configuration

Display Accounting configuration

5.4.2

Security configuration components 5.4.3

To view the security settings for a user

Security Access configuration

show commands for security access configuration

5.4.5

Cards, MDAs and ports configuration

display cards, MDAs and ports configuration 5.5

5.2. System Initialization troubleshooting 7750 SR hardware initialization takes place when a node is powered on or a running node is rebooted. By default, the system searches Compact Flash Slot #3 (cf3) for the boot.ldr file (also known as the bootstrap file). The boot.ldr file is the image that reads and executes the system initialization commands configured in the boot option file (bof.cfg). The default value to initially search for the boot.ldr file on cf3 cannot be modified. Once the system executes the boot.ldr file, it process the bof.cfg file which is stored on cf3, and by default, the system looks for this file on cf3. Troubleshooting Notes: If the bof.cfg file is not found, the system initialization will fail.

5.2.1. Boot Option File configuration The 7750 SR uses the Boot Option File (BOF) to start the system. The BOF file contains information to perform the following tasks: 1) Set up the CPM Ethernet port (speed, duplex, auto) 2) Create an IP address for the CPM Ethernet port 3) Create a static route for the CPM Ethernet port 4) Set the console port speed 5) Configure the DNS Domain name 6) Configure Primary, Secondary, Tertiary image location 7) Configure Primary, Secondary, Tertiary configuration location


62

8) Configure operational synchronization parameters between redundant SF/CPM cards 9) Configure persistence requirement It’s not necessary to have all the above information configured in a BOF. Following is an example of contents in a BOF file:

CLI Commands of BOF configuration

Task CLI commands

Display current system configuration

admin# display-config [detail|index]

info <detail>

show version

Display the BOF configuration

show bof [cflash-id|booted}

Modify a BOF configuration

bof# [no] address ip-addr/mask [active | standby] [no] autonegotiate no console-speed no dns-domain [no] primary-config file-url no primary-dns [no] primary-image file-url [no] secondary-config file-url no secondary-dns [no] secondary-image directory-url [no] static-route ip-prefix/mask next-hop ip-addr [no] tertiary-config file-url no tertiary-dns [no] tertiary-image file-url bof# save cflash-id

Save a BOF configuration

admin# save [file-url] [detail] [index]


63

Reboot admin# reboot

5.2.2. Troubleshooting notes on BOF configuration • The BOF file must specify at least one location for the runtime image. If a runtime

image cannot be loaded, the system will fail to start, and user intervention is required to correct the problem.

• If a configuration file cannot be found, the system is initialized with default

configuration settings and, the SNMP is shutdown. However the SNMP traps will continue to be issued. The system issues traps, log messages and console messages to advise the user. It requires a no shutdown snmp to reactivate full SNMP functionality.

• If there is no configuration file found in the BOF, any configuration change to the

system can not be saved and will be lost when the system is rebooted or shutdown. • Always be sure to save the BOF when any configuration change is made. • Persistence on/off: Persistence is required when the 7750 SR is managed by the 5620 SAM. If a node reboots with persistence turned on, it must locate the persistence index file and successfully process it before processing the system configuration file. If the index file cannot be processed for some reason, the system performs a SNMP shutdown. It requires a no shutdown snmp to reactivate full SNMP functionality.

5.2.3. Commands to check config file contents The 7750 SR file system is based on a DOS file system. In the 7750 SR routers, each control processor can have up to three compact flash devices (cf1:, cf2: or cf3:). The above device names are relative device names as they refer to the devices local to the control processor with the current console session. As in the DOS file system, the colon (“:”) at the end of the name indicates it is a device. The absolute device names for the compact flash devices are formed by appending, a dash and the slot control processor slot number (“A” and/or “B”) to the device number and preceding the colon, for example, “cf1-A:” is the absolute device name for compact flash device 1 in control processor slot A.


64

The following commands can be used to navigate file structure on a compact flash device and look at config file content.

Task CLI commands

1 To find the config file and the flash card (cf#) it is saved on

show bof

2 To find a file on the cf3 on the active SF/CPM card

file dir

file dir cf3:

3 To find a file on the cf3 of slot B (whether the SF/CPM in Slot B is active or standby)

file dir cf3-B:

4 To change directory (from one flash card (cf3-B) to another (cf3-A) )

file cd cf3-A:

5 To look at config file on cf3 file type file-url

(ex.

file type cf3:/log/log0202-20040714-190252)

Examples of output of the commands: 1. show bof SR12# show bof ============================================================================= BOF (Memory) ============================================================================= primary-image cf3:\images\R4 primary-config cf3:\SPIRIT_NCCHRL-X-R4.cfg address 138.120.199.117/24 active address 138.120.199.118/24 standby primary-dns 138.120.118.196 secondary-dns 138.120.118.198 dns-domain ca.newbridge.com static-route 128.251.10.0/24 next-hop 138.120.199.1 static-route 138.120.0.0/16 next-hop 138.120.199.1 autonegotiate duplex full speed 100 wait 3 persist on console-speed 115200 =============================================================================

2. file dir SR12# file dir


65

Volume in drive cf3 on slot A has no label. Directory of cf3:\ 09/08/2004 05:53a 1729589 boot.ldr 09/08/2004 08:51a 4110 bootlog.txt 09/06/2004 02:29a <DIR> 1.3.R4 09/04/2004 07:15a 118782 config.cfg 09/08/2004 07:34a 785 bof.cfg 09/08/2004 07:34a 785 bof.cfg.4 09/08/2004 07:34a 783 bof.cfg.1 09/08/2004 07:34a 783 bof.cfg.2 09/08/2004 07:34a 783 bof.cfg.3 05/30/2004 08:37a 126353 intial.cfg 09/06/2004 06:16a 66421 GAATLN-CORE01_TEST_X.cfg 09/06/2004 09:25a 25225 NCCHRL-CORE01_TEST_X.cfg 09/08/2004 07:34a 783 bof.cfg.5 09/06/2004 09:17a 87917 SCCLMA-CORE01_TEST_X.cfg 09/07/2004 06:56a 12474 SPIRIT_GAATLN-X-R4.cfg 09/08/2004 08:25a 102034 SPIRIT_NCCHRL-X-R4.cfg 09/08/2004 06:19a 28365 SPIRIT_SCCLMA-X-R4.cfg 09/08/2004 07:05a 13238 SPIRIT_GAATLN-R4.cfg 07/04/2004 03:52p 3799 bootlog_prev.txt 07/06/2004 07:16p 147041 james.cfg 09/08/2004 07:05a 27004 SPIRIT_NCCHRL-R4.cfg 09/08/2004 11:29a 28707 SPIRIT_SCCLMA-R4.cfg 09/08/2004 08:44a 1295 SPIRIT-SNMP-NDX.txt 07/08/2004 03:55p 89536 james_backup.cfg 09/08/2004 08:48a 1295 SPIRIT_NCCHRL-X-R4.ndx 09/08/2004 08:25a 24960 SPIRIT_NCCHRL-X-R4.cfg.1 09/08/2004 08:25a 24872 SPIRIT_NCCHRL-X-R4.cfg.2 07/08/2004 03:54p 89536 toroonxnec02.cfg 07/08/2004 03:54p 89536 toroonxnec02.cfg.1 07/08/2004 03:54p 89507 toroonxnec02.cfg.2 09/08/2004 08:25a 24872 SPIRIT_NCCHRL-X-R4.cfg.3 09/08/2004 08:25a 26563 SPIRIT_NCCHRL-X-R4.cfg.4 09/01/2004 05:52a <DIR> images 09/01/2004 04:41a 118771 marcel.config.cfg 09/01/2004 04:59a 118771 marcel.R2.config.cfg 33 File(s) 3225275 bytes. 2 Dir(s) 99076096 bytes free.

3. file dir cf3-B: SR12# file dir cf3-B: Volume in drive cf3 on slot B has no label. Directory of cf3:\ 09/06/2004 07:32a 118782 config.cfg 07/22/2004 09:49p 2070 bootlog.txt 07/22/2004 06:55p 1729589 boot.ldr 07/22/2004 02:34p <DIR> 1.3.R4 07/21/2004 08:51p <DIR> images 07/22/2004 06:54p 784 bof.cfg.bak 07/22/2004 06:54p 28365 SCCLMA-CORE01_TEST_X.cfg 07/06/2004 08:16p 147041 james.cfg


66

09/08/2004 11:34a 785 bof.cfg 07/21/2004 05:09p 12474 SPIRIT_GAATLN-X-R4.cfg 07/22/2004 07:32p 510 bof.cfg.2 06/11/2004 03:10p 15445 erica.cfg 09/08/2004 12:52p 102034 SPIRIT_NCCHRL-X-R4.cfg 05/10/2004 06:23p 2070 bootlog_prev.txt 09/08/2004 11:01a 28365 SPIRIT_SCCLMA-X-R4.cfg 07/22/2004 07:32p 779 bof.cfg.1 07/22/2004 06:55p 1824066 boot.ldr.bak 07/22/2004 07:32p 510 bof.cfg.3 07/22/2004 06:54p 87917 SCCLMA-CORE01_TEST_X.cfg.bak 09/08/2004 11:16a 27004 SPIRIT_GAATLN-R4.cfg 09/08/2004 10:16a 29376 SPIRIT_SCCLMA-X-R4.cfg.bak 07/22/2004 07:03p 1738 SPIRIT_SCCLMA-X-R4.ndx 06/29/2004 10:45p 17012 metro.cfg 06/29/2004 11:20p 20137 metro_colo_1.cfg 06/30/2004 09:19p 45393 ylcolo.cfg 06/30/2004 10:03p 20359 ylconfig.cfg 06/30/2004 09:57p 47759 ylcomplete.cfg 09/08/2004 12:51p 24960 SPIRIT_NCCHRL-X-R4.cfg.bak 09/08/2004 11:23a 28740 SPIRIT_NCCHRL-R4.cfg 09/08/2004 11:16a 13238 SPIRIT_GAATLN-R4.cfg.bak 09/08/2004 11:34a 26563 SPIRIT_SCCLMA-R4.cfg 09/08/2004 11:23a 27004 SPIRIT_NCCHRL-R4.cfg.bak 09/08/2004 11:34a 28707 SPIRIT_SCCLMA-R4.cfg.bak 09/08/2004 12:52p 1295 SPIRIT_NCCHRL-X-R4.ndx

09/08/2004 12:52p 2529 SPIRIT_NCCHRL-X-R4.ndx.bak 33 File(s) 4463400 bytes. 2 Dir(s) 23481344 bytes free.

4. file cd cf3-A: SR12# file cd cf3-A: SR12# file dir Volume in drive cf3 on slot A has no label. Directory of cf3:\ 09/08/2004 05:53a 1729589 boot.ldr 09/08/2004 08:51a 4110 bootlog.txt 09/06/2004 02:29a <DIR> 1.3.R4 09/04/2004 07:15a 118782 config.cfg 09/08/2004 07:34a 785 bof.cfg 09/08/2004 07:34a 785 bof.cfg.4 09/08/2004 07:34a 783 bof.cfg.1 09/08/2004 07:34a 783 bof.cfg.2 09/08/2004 07:34a 783 bof.cfg.3 05/30/2004 08:37a 126353 intial.cfg 09/06/2004 06:16a 66421 GAATLN-CORE01_TEST_X.cfg 09/06/2004 09:25a 25225 NCCHRL-CORE01_TEST_X.cfg 09/08/2004 07:34a 783 bof.cfg.5 09/06/2004 09:17a 87917 SCCLMA-CORE01_TEST_X.cfg 09/07/2004 06:56a 12474 SPIRIT_GAATLN-X-R4.cfg 09/08/2004 08:25a 102034 SPIRIT_NCCHRL-X-R4.cfg 09/08/2004 06:19a 28365 SPIRIT_SCCLMA-X-R4.cfg Press any key to continue (Q to quit) . .


67

.

5. file type file-url SR12# file type config.cfg # TiMOS-C-2.0.R4 cpm/hops ALCATEL SR 7750 Copyright (c) 2000-2004 Alcatel. # All rights reserved. All use subject to applicable license agreements. # Built on Fri Jul 9 13:18:19 PST 2004 by builder in /rel2.0/b4/R4/panos/main # Generated SAT SEP 04 12:15:15 2004 UTC exit all configure #------------------------------------------ echo "System Configuration" #------------------------------------------ system name "TOROONXNEC14" no contact no location no clli-code no coordinates no config-backup no boot-good-exec no boot-bad-exec power-supply 1 dc power-supply 2 none lacp-system-priority 32768 synchronize config snmp engineID "0000197f000000000003fa0b" packet-size 9216 general-port 161 no shutdown exit login-control ftp inbound-max-sessions 3 . . .

5.3. Verify System management configuration

Task CLI commands Section

Display system information

show system information

show uptime

show version

5.3.1


68

Display SF/CPMs redundancy configuration

show system synchronization

show card

Automatically synchronize two SF/CPMs

config>system

synchronize [boot-env|config]

Manually synchronize two SF/CPMs

admin# synchronize config

5.3.2

Timing configuration show system sync-if-timing

Change a timing reference input mode to be revertive or non-revertive

config>system>sync-if-timing# revert

Force the system timing output to use a specific reference

debug>sync-if-timing# force-reference

Warning: this command is only used for debugging, configuration will not be saved between reboots.

5.3.3

SNTP configuration show system sntp 5.3.4

CPU utilization show system cpu

Memory show system memory-pools

5.3.1. Display system information

CLI Syntax: show system information


69

5.3.2. Verify Synchronization and Redundancy 7750 SR routers supporting redundancy (on 7750 SR-7 & SR-12 models) use a 1:1 redundancy scheme. Redundancy methods facilitate system synchronization between the active and standby Control Processor Modules (CPMs) so they maintain identical operational parameters to prevent inconsistencies in the event of a CPM failure. Although software configurations and images can be copied or downloaded from remote locations, synchronization can only occur locally between compact flash drives (cf1:, cf2:, and cf3:). Synchronization can occur either automatically or manually. When automatic system synchronization is enabled for an entity, any save or delete file operations configured on the primary, secondary or tertiary choices on the active CPM file system are mirrored in the standby CPM file system.

CLI Syntax: show system synchronization


70

Automatic synchronization Automatic synchronization is disabled by default. To enable automatic synchronization, the config>system>synchronization command must be specified with either the boot-env parameter or the config parameter. When the boot-env parameter is specified, the BOF, boot.ldr, config, and image files are automatically synchronized. When the config parameter is specified, only the config files are automatically synchronized. Automatic synchronization also occurs whenever the BOF is modified and when an admin>save command is entered with no filename specified. CLI Syntax: config>system

synchronize [boot-env|config]

Manual synchronization To execute synchronization manually, the admin>synchronization command must be entered with the boot-env parameter or the config parameter. When the boot-env parameter is specified, the BOF, boot.ldr, config, and image files are synchronized. When the config parameter is specified, only the config files are synchronized. CLI Syntax: admin>synchronize {boot-env|config} Example: admin# synchronize config The following shows the output which displays during a manual synchronization:

5.3.3. Verify timing configuration

CLI syntax: show system sync-if-timing


71

Using the Revert Command The revert command allows the clock to revert to a higher priority reference if the current reference goes offline or becomes unstable. When the failed reference becomes operational, it is eligible for selection. When mode is non-revertive, a failed clock source is not selected again. CLI Syntax: config>system>sync-if-timing# revert Forcing a Specific Reference You can force the system synchronous timing input to use a specific reference. NOTE: The debug sync-if-timing force-reference command should only be used to test and debug problems. Once the system timing reference input has been forced, it will not revert back to another reference unless explicitly reconfigured. When the command is executed, the current system synchronous timing output is immediately referenced from the specified reference input. If the specified input is not available (shutdown), or in a disqualified state, the timing output will enter a holdover state based on the previous input reference.

Debug configurations are not saved between reboots.


72

CLI Syntax: debug>sync-if-timing# force-reference {ref1 | ref2 | bits}

5.3.4. Verify SNTP configuration SNTP is a compact, client-only version of the NTP. SNTP can only receive the time from SNTP/ NTP servers; it cannot be used to provide time services to other systems. SNTP can be configured in either broadcast or unicast client mode.

Sample output to show current setting:

5.4. Security Access configuration

The 7750 SR can be accessed in three ways:

• CLI via the console, Telnet, and FTP

• Secure Shell (SSH)/secure Copy Protocol (SCP)

• SNMP – the 7750 is fully compliant with SNMPv3 and backward compliant with SNMPv1 and v2c.

Authentication is supported on local access, RADIUS, or TACACS+.

Authorization is supported on local access, RADIUS, or TACACS+.

Accounting is supported only on RADIUS and TACACS+.

5.4.1. Authentication, Authorization and Accounting The 7750 SR uses authentication, authorization, and accounting (AAA) to monitor and control network access to the router. Network security is implemented in a step-by-step process, starting with authentication, then authorization and may also include accounting. The first step, authentication, validates a user’s name and password. The second step is authorization, which allows the user to access and execute commands at various command levels based on profiles assigned to the user.


73

Another step, accounting, keeps track of the activity of a user who has accessed the network. The type of accounting information recorded can include a history of the commands executed, the amount of time spent in the session, the services accessed, and the data transfer size during the session. The accounting data can then be used to analyze trends, and also for billing and auditing purposes.

7750 SR OS supports the following security features: • RADIUS can be used for authentication, authorization, and accounting. • TACACS+ can be used for authentication, authorization, and accounting. • Local security can be implemented for authentication and authorization.

You can select one or more of the above security methods and configure the order in which the security methods are applied. Authentication Authentication validates a user name and password combination when a user attempts to log in. When a user attempts to log in through the console, Telnet, SSH, SCP, or FTP, the 7750 SR client sends an access request to a RADIUS, TACACS+, or local database. Transactions between the client and a RADIUS server are authenticated through the use of a shared secret. The secret is never transmitted over the network. User passwords are sent encrypted between the client and RADIUS server which prevents someone snooping on an insecure network to learn password information. If the RADIUS server does not respond within a specified time, the router issues the access request to the next of the configured servers. Each RADIUS server must be configured identically to guarantee consistent results. If any RADIUS server rejects the authentication request, it sends an access reject message to the router. In this case, no access request is issued to any other RADIUS servers. However, if other authentication methods such as TACACS+ and/or local are configured, then these methods are attempted. If no other authentication methods are configured, or all methods reject the authentication request, then access is denied. The user login is successful when the RADIUS server accepts the authentication request and responds to the router with an access accept message. Implementing authentication without authorization for the 7750 SR routers does not require the configuration of VSAs (Vendor Specific Attributes) on the RADIUS server. However, users, user access permissions, and command authorization profiles must be configured on each router. Any combination of these authentication methods can be configured to control network access from a 7750 SR router:

• Local Authentication • RADIUS Authentication • TACACS+ Authentication


74

Authorization 7750 SR routers support local, RADIUS, and TACACS+ authorization to control the actions of specific users by applying a profile based on user name and password configurations once network access is granted. The profiles are configured on locally as well as VSAs (Vendor Specific Attributes ) on the RADIUS server. Once a user has been authenticated using RADIUS (or another method), the 7750 SR router perform authorization if configured to do so. The RADIUS server can be used to:

• Download the user profile to the 7750 SR router • Send the profile name that the node should apply to the 7750 SR router.

Profiles consist of a suite of commands that the user is allowed or not allowed to execute. When a user issues a command, the authorization server looks at the command and the user information and compares it with the commands in the profile. If the user is authorized to issue the command, the command is executed. If the user is not authorized to issue the command, then the command is not executed. Profiles must be created on each 7750 SR router and should be identical for consistent results. If the profile is not present, then access is denied.

Accounting When enabled, RADIUS accounting sends command line accounting from the 7750 SR router to the RADIUS server. The router sends accounting records using UDP packets on port 1813 (decimal). The router issues an accounting request packet for each event requiring the activity to be recorded by the RADIUS server. The RADIUS server acknowledges each accounting request by sending an accounting response after it has processed the accounting request. If no response is received in the time defined in the timeout parameter, the accounting request must be retransmitted until the configured retry count is exhausted. A trap is issued to alert the NMS (or trap receiver) that the server is unresponsive. The router issues the accounting request to the next configured RADIUS server (up to 5). User passwords and authentication keys of any type are never transmitted as part of the accounting request.

5.4.2. How AAA is configured

The following tables show how AAA is configured in different scenarios respectively. They also provide the index of where to troubleshoot a configuration under a certain circumstance.

Table 12: Configuring Authentication

Notes: RADIUS 1* - for RADIUS authentication only

RADIUS 2* - for RADIUS authentication (with authorization)


75

Components be configured Local RADIUS 1*

RADIUS 2*

TACACS+

Section

Password management parameters

• 5.4.3.2

Profiles • • • 5.4.3.3

User access parameters • • • 5.4.3.4

RADIUS Authentication • • 5.4.3.5

RADIUS Authorization • 5.4.3.6

TACACS+ Authentication • 5.4.3.9

Table 13: Configuring Authorization

Notes: RADIUS 1* - for RADIUS authorization only (without authentication)

RADIUS 2* - for RADIUS authorization (with authentication)

TACACS+ 1* - for TACACS+ authorization only

TACACS+ 2* - for TACACS+ authorization with authentication

Components be configured Local RADIUS 1*

RADIUS 2*

TACACS+ 1*

TACACS+ 2*

Section

Profiles • • • 5.4.3.3

User access parameters • 5.4.3.4

RADIUS Authentication • 5.4.3.5

RADIUS Authorization • • 5.4.3.6

RADIUS VSA • • 5.4.3.7

TACACS+ Authentication • 5.4.3.9

TACACS+ Authorization • • 5.4.3.10


76

Table 14: Configuring Accounting

Components be configured Local RADIUS TACACS+ Section

RADIUS Accounting N/A • 5.4.3.8

TACACS+ Accounting N/A • 5.4.3.11

5.4.3. Security Configuration Components To implement security features, configure the following components:

• Management access filters (optional) • Profiles • User access parameters • Password management parameters • Enable RADIUS and/or TACACS+

o One to five RADIUS and/or TACACS+ servers o RADIUS and/or TACACS+ parameters

The following sub-sections describe the details of the configuration of each component.

5.4.3.1 Configuring Management access filters Creating and implementing management access filters is optional. Management access filters control all traffic going in and out of the CPM, including all routing protocols. The filters can be used to restrict management of the 7750 SR router by other nodes outside either specific (sub)networks or through designated ports. By default, there are no filters associated with security options. The management access filter and entries must be explicitly created on each router. The 7750 SR OS implementation exits the filter when the first match is found and execute the actions according to the specified action. For this reason, entries must be sequenced correctly from most to least explicit. An entry may not have any match criteria defined (in which case, everything matches) but must have at least the keyword action to be considered complete. Entries without the action keyword are considered incomplete and will be rendered inactive. Use the following CLI commands to configure a management access filter. This example only accepts packets matching the criteria specified in entries 1 and 2. Non-matching packets are denied.


77

The following displays an example of the management access filter command usage.

The following example displays the management access filter configuration:

5.4.3.2 Configuring Password management parameters Password management parameters consists of defining aging, the authentication order and authentication methods, password length and complexity, as well as the number of attempts a user can enter a password. Depending on the authentication requirements, password parameters are configured locally. Use the following CLI commands to configure password support:


78

The following displays an example of the password command usage.

The following example displays the password configuration:

5.4.3.3 Configuring profiles Profiles are used to deny or permit access to a hierarchical branch or specific commands. Profiles are referenced in a user configuration. A maximum of sixteen user profiles can be defined. A user can participate in up to sixteen profiles. Depending on the the authorization requirements, passwords are configured locally or on the RADIUS server. Use the following CLI commands to configure user profiles:

The following displays an example of the user profile command usage.


79

The following example displays the user profile output:

5.4.3.4 Configuring User access parameters Configure access parameters for individual users. For user, define the login name for the user and, optionally, information that identifies the user. Use the following CLI commands to configure RADIUS support:


80

The following example displays the user configuration:

5.4.3.5 Configuring RADIUS Authentication RADIUS is disabled by default and must be explicitly enabled. The mandatory commands to enable RADIUS on the local router are radius and server index address ip-addr secret key. The other commands are optional. The server command adds a RADIUS server and configures the RADIUS server’s IP address, index, and key values. The index determines the sequence in which the servers are queried for authentication requests. On the local router, use the following CLI commands to configure RADIUS authentication:


81

The following example displays the CLI syntax usage:

The following example displays the RADIUS authentication configuration:

5.4.3.6 Configuring RADIUS Authorization In order for RADIUS authorization to function, RADIUS authentication must be enabled first. In addition to the local configuration requirements, VSAs must be configured on the RADIUS server. On the local router, use the following CLI commands to configure RADIUS authorization:


The following example displays the RADIUS authorization configuration:


82

5.4.3.7 Configuring VSA when RADIUS Authorization is enabled 7750 SR OS software supports the configuration of Alcatel-specific RADIUS attributes. These attributes are known as Vendor-Specific Attributes (VSAs) and are discussed in RFC 2138. VSAs must be configured when RADIUS authorization is enabled. It is up to the vendor to specify the format of their VSA. The attribute-specific field is dependent on the vendor’s definition of that attribute. The Alcatel-defined attributes are encapsulated in a RADIUS vendor-specific attribute with the vendor ID field set to 6527, the vendor ID number. The following RADIUS vendor-specific attributes (VSAs) are supported by Alcatel. • timetra-access <ftp> <console> <both> — This is a mandatory command must be configured. This command specifies if the user has FTP and /or console (serial port, Telnet, and SSH) access. • timetra-profile <profile-name> — When configuring this VSA for a user, it is assumed that the user profiles are configured on the local 7750 SR router and the following applies for local and remote authentication: 1. The authentication-order parameters configured on the router must include the local keyword. 2. The user name may or may not be configured on the 7750 SR router. 3. The user must be authenticated by the RADIUS server 4. Up to 8 valid profiles can exist on the router for a user. The sequence in which the profiles are specified is relevant. The most explicit matching criteria must be ordered first. The process stops when the first complete match is found. If all the above mentioned conditions are not met, then access to the router is denied and a failed login event/trap is written to the security log. • timetra-default-action <permit-all|deny-all|none> — This is a mandatory command must be configured. This command specifies the default action when the user has entered a command and no entry configured in the timetra-cmd VSA for the user resulted in a match condition.


83

• timetra-cmd <match-string> — Configures a command or command subtree as the scope for the match condition. The command and all subordinate commands in subordinate command levels are specified. Configure from most specific to least specific. The 7750 SR OS implementation exits on the first match, subordinate levels cannot be modified with subsequent action commands. Subordinate level VSAs must be entered prior to this entry to be effective. All commands at and below the hierarchy level of the matched command are subject to the timetra-action VSA. Multiple match-strings can be entered in a single timetra-cmd VSA. Match strings must be semicolon (;) separated (maximum string length is 254 characters). One or more timetra-cmd VSAs can be entered followed by a single timetra-action VSA. • timetra-action <deny|permit> — Causes the permit or deny action to be applied to all match strings specified since the last timetra-action VSA. • timetra-home-directory <home-directory string> — Specifies the home directory that applies for the FTP and CLI user. If this VSA is not configured, the home directory is Compact Flash slot 1 (cf1:). • timetra-restrict-to-home-directory <true|false> — Specifies if user access is limited to their home directory (and directories and files subordinate to their home directory). If this VSA is not configured the user is allowed to access the entire file system. • timetra-login-exec <login-exec-string> — Specifies the login exec file that is executed when the user login is successful. If this VSA is not configured no login exec file is applied. If no VSAs are configured for a user, then the following applies: 1. The password authentication-order command on the 7750 SR router must include local. 2. The user name must be configured on the 7750 SR router. 3. The user must be successfully be authenticated by the RADIUS server 4. A valid profile must exist on the 7750 SR router for this user. If all conditions listed above are not met, then access to the 7750 SR router is denied and a failed login event/trap is written to the security log. Sample User (VSA) Configuration The following example displays a user-specific VSA configuration. This configuration shows attributes for users named ruser1 and ruser2.


84

The following example shows that user ruser1 is granted console access. ruser1’s home directory is in compact flash slot 3 and is limited to the home directory. The default action permits all packets when matching conditions are not met. The timetra-cmd parameters allow the user to use the tools;telnet;configure system security commands. Matching strings specified in the timetra-action command are denied for this user. The user ruser2 is granted FTP access.The default action denies all packets when matching conditions are not met. The timetra-cmd parameters allow the user to use the configure, show, and debug commands. Matching strings specified in the timetraaction command are permitted for this user.

Timetra Dictionary


85

5.4.3.8 Configuring RADIUS Accounting On the local router, use the following CLI commands to configure RADIUS accounting:


The following example displays the RADIUS accounting configuration:


86

5.4.3.9 Enabling TACACS+ Authentication To use TACACS+ authentication on the router, configure one or more TACACS+ servers on the network. Use the following CLI commands to configure profiles:

The following example is configured in the config>system context:

The following example displays the TACACS+ authentication configuration:


87

5.4.3.10 Configuring TACACS+ Authorization In order for TACACS+ authorization to function, TACACS+ authentication must be enabled first. On the local router, use the following CLI commands to configure RADIUS authorization:


The following example displays the TACACS+ authorization configuration:

5.4.3.11 Configuring TACACS+ Accounting On the local router, use the following CLI commands to configure TACACS+ accounting:



88

The following example displays the TACACS+ accounting configuration:

5.4.3.12 Enabling SSH The ssh-server command starts the SSH server. There are no configurable parameters in the SSH context. To enable SSH, enter the following CLI syntax. CLI Syntax: config>system>security

ssh-server

Example: config>system>security#ssh-server The following example displays the SSH server configuration:

5.4.3.13 Configuring Login controls Configure login control parameters for console, Telnet, and FTP sessions. To configure login controls, enter the following CLI syntax.


89

The following example displays the login control configuration:

5.4.4. SNMP security configuration

5.4.4.1 SNMP overview SNMP Architecture The Network Management System (NMS) is comprised of two elements: managers and agents. The manager is the entity through which network management tasks are facilitated. Agents interface managed objects. Managed devices, such as bridges, hubs, routers, and network servers can contain managed objects. A managed object can be a configuration attribute, performance statistic, or control action that is directly related to the operation of a device. Managed devices collect and store management information and use Simple Network Management Protocol (SNMP). SNMP is an application-layer protocol that provides a message format to


90

facilitate communication between SNMP managers and agents. SNMP provides a standard framework to monitor and manage devices in a network from a central location. An SNMP manager controls and monitors the activities of network hosts which use SNMP. An SNMP manager can obtain (get) a value from an SNMP agent or store (set) a value in the agent. The manager uses definitions in the management information base (MIB) to perform operations on the managed device such as retrieving values from variables or blocks of data, replying to requests, and processing traps. Between the SNMP agent and the SNMP manager the following actions can occur:

• The manager can get information from the agent. • The manager can set the value of a MIB object that is controlled by an agent. • The agent can send traps to notify the manager of significant events that occur on the 7750 SR router.

SNMP Versions The agent supports multiple versions of the SNMP protocol. • SNMP Version 1 (SNMPv1) is the original Internet-standard network management framework. SNMPv1 uses a community string match for authentication. • The 7750 SR OS implementation uses SNMPv2c, the community-based administrative framework for SNMPv2. SNMPv2c uses a community string match for authentication. • In SNMP Version 3 (SNMPv3), USM defines the user authentication and encryption features. View Access Control MIB (VACM) defines the user access control features. The SNMP-COMMUNITY-MIB is used to associate SNMPv1/SNMPv2c community strings with SNMPv3 VACM access control. SNMPv3 uses a username match for authentication. Management Information Access Control By default, the 7750 SR OS implementation of SNMP uses SNMPv3. SNMPv3 incorporates security model and security level features. A security model is the authentication type for the group and the security level is the permitted level of security within a security model. The combination of the security level and security model determines which security mechanism handles an SNMP packet. To implement SNMPv1 and SNMPv2c configurations, several access groups are predefined. These access groups provide standard read-only, read-write, and read-write-all access groups and views that can simply be assigned community strings. In order to implement SNMP with security features, security models, security levels, and USM communities must be explicitly configured. Optionally, additional views which specify more specific OIDs (MIB objects in the subtree) can be configured.


91

Access to the management information in as SNMPv1/SNMPv2c agent is controlled by the inclusion of a community name string in the SNMP request. The community defines the subset of the agent’s managed objects can be accessed by the requester. It also defines what type of access is allowed: read-only or read-write. The use of community strings provide minimal security and context checking for both agents and managers that receive requests and initiate trap operations. A community string is a text string that acts like a password to permit access to the agent on the 7750 SR router. Alcatel’s implementation of SNMP has defined three levels of community-named access: • Read-Only permission — Grants only read access to objects in the MIB, except security objects. • Read-Write permission — Grants read and write access to all objects in the MIB, except security objects. • Read-Write-All permission — Grants read and write access to all objects in the MIB, including security objects. User-based Security Model Community Strings User-based security model (USM) community strings associates a community string with an SNMPv3 access group and its view. The access granted with a community string is restricted to the scope of the configured group. Views Views control the access to a managed object. The total MIB of a 7750 SR router can be viewed as a hierarchical tree. When a view is created, either the entire tree or a portion of the tree can be specified and made available to a user to manage the objects contained in the subtree. Object identifiers (OIDs) uniquely identify managed objects. A view defines the type of operations for the view such as read, write, or notify. OIDs are organized in a hierarchical tree with specific values assigned to different organizations. A view defines a subset of the agent’s managed objects controlled by the access rules associated with that view. Pre-defined views are available that are particularly useful when configuring SNMPv1 and SNMPv2c. The Alcatel SNMP agent associates SNMPv1 and SNMPv2c community strings with a SNMPv3 view. Access Groups Access groups associate a user group and a security model to the views the group can access. An access group is defined by a unique combination of a group name, security model (SNMPv1, SNMPv2c, or SNMPv3), and security level (no-authorization-no privacy, authorization-no-privacy, or privacy).


92

An access group, in essence, is a template which defines a combination of access privileges and views. A group can be associated to one or more network users to control their access privileges and views. Additional access parameters must be explicitly configured if the preconfigured access groups and views for SNMPv1 and SNMPv2c do not meet your security requirements. Users By default, authentication and encryption parameters are not configured. Authentication parameters which a user must use in order to be validated by the 7750 SR device can be modified. SNMP authentication allows the device to validate the managing node that issued the SNMP message and determine if the message has been tampered with. User access and authentication privileges must be explicitly configured. In a user configuration, a user is associated with an access group, which is a collection of users who have common access privileges and views (see Access Groups).

5.4.4.2 Which SNMP version to use SNMPv1 and SNMPv2c do not provide security, authentication, or encryption. Without authentication, a non authorized user could perform SNMP network management functions and eavesdrop on management information as it passes from system to system. Many SNMPv1 and SNMPv2c implementations are restricted read-only access, which, in turn, reduces the effectiveness of a network monitor in which network control applications cannot be supported. To implement SNMPv3, an authentication and encryption method must be assigned to a user in order to be validated by the 7750 SR device. SNMP authentication allows the router to validate the managing node that issued the SNMP message and determine if the message was tampered with. Figure 15 depicts the configuration requirements to implement SNMPv1/SNMPv2c, and SNMPv3.


93

Figure 15: SNMPv1 and SNMPv2c Configuration and Implementation Flow

5.4.4.3 SNMP security configuration components Figure 16displays the major components to configure SNMP.

Figure 16: SNMP Configuration Components

• Community — The community string is an access environment for a group of network management systems. The string acts like a password to control client access to the server. The access granted with a community string depends on the read or read-write parameters. • USM Community — The USM community string associates an SNMPv1/SNMPv2 community string with an access group and a view. • View — Views control access to a managed object.


94

• Access group — The access group creates an association between a group of users, a security model, and the views the group can access. • User — Users are associated with an access group and, therefore, share common security models and access views.

5.4.4.4 Commands displaying SNMP security configuration

Task CLI commands Display the SNMP configuration and statistics

show system information

List SNMP communities and characteristics.

show system security communities

List one or all views and permissions in the MIB-OID tree.

show system security view [view-name] [detail]

Display access-group information

show system security access-group [group-name]

Display user information

show system security user [user-id] [detail]

5.4.5. User Access failure troubleshooting

If a user fails to be authenticated, he/she is NOT allowed to login to the system.

The authorization applies to a user who has passed the authentication, but he/she is NOT allowed to execute a certain command level.

Only the administrative level user can modify the other users’ profiles.

Check the user access settings for that user, modify if configuration is improper. 1. CLI commands to view the security settings for a user:

show users

configure system security user <user-name>

info detail

2. CLI commands to view/configure user access parameters for a specific user: Task CLI commands Grant or deny user permission for FTP, SNMP, or console access.

configure system security user# access

info detail


95

Specify the user password for console and FTP access.

configure system security user#

password

Configure user profile membership for the console (either Telnet or a CPM serial port user).

configure system security user# console

info detail

3. Other show commands Display console user login and connection information

show users

Display system login authentication configuration and statistics

show system security authentication [statistics]

Display management access control filter information

show system security management-access-filter [entry-num]

Display configured password options

show system security password-options

Display user profile information.

show system security profile [user-profile-name]

Display the SSH server state and the SSH connections

show system security ssh

Display user registration information

show system security user [userid] [detail]

5.5. Verify Event & Accounting logs configuration

7750 SR supports two types of logging, event logging and accounting logging.

Event logging controls the generation, dissemination and recording of system events for monitoring status and troubleshooting faults within the system. Refer to Section 3.1.1 for Event logging overview.

5.5.1. Accounting logging Overview An event log within 7750 SR OS associates the event sources with logging destinations. Examples of logging destinations include, all console sessions, a specific console session, memory logs, file


96

destinations, SNMP trap groups and syslog destinations. A log filter policy can be associated with the event log to control which events will be logged in the event log based on combinations of application, severity, event ID range and the subject of the event. The 7750 SR accounting logs collect comprehensive accounting statistics to support a variety of billing models. The 7750 SR collects accounting data on services and network ports on a per service class basis. In addition to gathering information critical for service billing, accounting records can be analyzed to provide insight about customer service trends for potential service revenue opportunities. Accounting statistics on network ports can be used to track link utilization and network traffic pattern trends. This information is valuable for traffic engineering and capacity planning within the network core. Accounting statistics are collected according to the parameters defined within the context of an accounting policy. Accounting policies are applied to customer Service Access Points (SAPs) and network ports. Accounting statistics are collected by counters for individual service queues defined on the customer’s SAP or by the counters within forwarding class (FC) queues defined on the network ports. The type of record defined within the accounting policy determines where a policy is applied, what statistics are collected and time interval at which to collect statistics. The only supported destination for an accounting log is a compact flash system device (cf1 or cf2). Accounting data is stored within a standard directory structure on the device in compressed XML format.

Accounting log files Before an accounting policy can be created a target log file must be created to collect the accounting records. The files are stored in system memory on a compact flash (cf1 or cf2) in a compressed (tar) XML format and can be retrieved using FTP or SCP. A file ID can only be assigned to either one event log ID or one accounting log. When a policy has been created and applied to a service or network port, the accounting file is stored on the compact flash in a compressed XML file format. The 7750 SR creates two directories on the compact flash to store the files. The following output displays a directory named act-collect that holds accounting files that are open and actively collecting statistics. The directory named act stores the files that have been closed and are awaiting retrieval.


97

Accounting files always have the prefix act followed by the accounting policy ID, log ID and timestamp. The accounting log file naming convention and log file destination properties like rollover and retention are similar with an event log file.

Accounting Records An accounting policy must define a record name and collection interval. Only one record name can be configured per accounting policy. The record name, sub-record types, and default collection period for service and network accounting policies are shown below.

Table 15: Accounting Record Name and Collection Periods

When creating accounting policies, one service accounting policy and one network accounting policy can be defined as default. If statistics collection is enabled on a SAP or network port and no accounting policy is applied, then the respective default policy is used. If no default policy is defined, then no statistics are collected unless a specifically defined accounting policy is applied. Each accounting record name is composed of one or more sub-records which is in turn composed of multiple fields. Design Considerations


98

The 7750 SR has ample resources to support large scale accounting policy deployments. When preparing for an accounting policy deployment, verify that data collection, file rollover, and file retention intervals are properly tuned for the amount of statistics to be collected. If the accounting policy collection interval is too brief there may be insufficient time to store the data from all the services within the specified interval. If that is the case, some records may be lost or incomplete. Interval time, record types, and number of services using an accounting policy are all factors that should be considered when implementing accounting policies. The rollover and retention intervals on the log files and the frequency of file retrieval must also be considered when designing accounting policy deployments. The amount of data stored depends on the type of record collected, the number of services that are collecting statistics, and the collection interval that is used. For example, with a 1GB CF and using the default collection interval, the system is expected to hold 48 hours worth of billing information.

5.5.2. Verifying the logging configurations

The following table provides the commonly used CLI commands to verify the existing logging configurations.

Information to view show commands Display a list of all application names that can be used in event-control and filter commands

show log applications

Display event control settings for events including whether the event is suppressed or generated and the severity level for the event

show log event-control [application [event-name | event-number]]

Display event file log information

show log file-id [file-id]

Display event log filter policy information

show log filter-id [filter-id]

Show log collector statistics for the main, security, change and debug log collectors

show log log-collector

Display an event log summary with settings and statistics or the contents of a specific log file, SNMP log, or memory log

show log log-id [log-id] [severity severity-level] [application application] [sequence from-seq [toseq]] [count number] [subject subject] [ascending | descending]

configure log log-id [log-id] <enter>

log-id# info detail

Display SNMP trap group configuration information

show log snmp-trap-group [log-id]


99

Display syslog event log destination summary information or detailed information on a specific syslog destination

show log syslog [syslog-id]

Display accounting policy information

show log accounting-policy [policy-id] [access | network]

Display accounting policy record names

show log accounting-records


100

6. Common troubleshooting scenarios

This section covers some common troubleshooting scenarios that might happen in a 7750 SR network.

6.1. Layer 1 & Layer 2 Problems

This section describes methods/commands that can be used to troubleshoot a Layer 1 or layer 2 (i.e. IOM, MDA and port level) problem of 7750 SRs. More details of how to verify hardware operational status are described in Section 4.

6.1.1. How to show Layer 1 & Layer 2 alarms

7750 SR has two default memory logs (Log-id 99 & 100) containing all the events from the “main” application. All severity levels of alarms are recorded in log-id 99, where log-id 100 only contains serious errors.

There are several ways to view the alarms of a specific subject, such as alarms related to a particular port. One method is to create a new log that only monitors the specific subject. Refer to 7750_SR_OS_System_Guide_2.0.pdf for more details of how to configure a log.

Another much simpler way is to view the specific subject in the default log-id 99. The following shows what command(s) to use for Layer 1& 2 alarms.

What To Check CLI Command

Show alarms of a particular port (ex. port 1/1/1)

show log log-id 99 subject 1/1/1

Show alarms related to the chassis

show log log-id 99 application chassis

Show alarms of a particular IOM

(ex. IOM Slot #1)

show log log-id 99 subject “Card 1”

Show alarms of a SF/CPM

(ex. SF/CPM Slot #A)

show log log-id 99 subject “Card A”

Show alarms of a particular MDA

(ex. MDA 1/1)

show log log-id 99 subject “Mda 1/1”

Note: All the commands are Case Sensitive.

6.1.2. Verify cards, MDAs and ports configuration


101

The following CLI commands are commonly used for checking the detailed configuration of cards, MDAs or ports. Refer to Section 4 for more information on hardware operational status.


Chassis

configuration & status

show chassis

show chassis environment

show chassis power-supply

IOM or SF/CPM


show card

show card <A/B> detail

show card <slot-number> detail

MDA


show mda

show mda detail

port


show port

show port <slot/mda/port>

show port <slot/mda/port[.sonet-sdh-index]>

show port <port-id> detail

show port <port-id> ppp [detail]

Link Aggregation Group (LAG)

show lag <lag-id>

show lag <lag-id> detail

Display logical interfaces associated with a port

show port <slot/mda/port> associations

6.1.3. How to show or clear statistics on a port or a LAG or a SAP


show statistics of a port show port <slot/mda/port> count


102

show statistics of a LAG show lag <lag-id> detail statistics

show counters of a SAP show service id <service-id> sap <port-id[:encap-val]> detail

clear counters of a port clear port < slot/mda/port > statistics

clear counters of a LAG clear lag <lag-id> statistics

clear counters of a SAP clear service statistics sap <port-id[:encap-val]> counters

6.1.4. How to show or modify the operational status of a port

Troubleshooting note: Ports by default are administratively down.

If a port is correctly configured but not up, most likely the port is administratively down.


To display the administrative status of a

port

show port <slot/mda/port>

To modify the administrative status of a

port

config port <slot/mda/port>

[no] shutdown

6.1.5. How to loop ports

Ethernet ports:

You can NOT loop Ethernet ports using CLI commands.

SONET/SDH ports:

You can use CLI command to loopback a SONET/SDH port.

NOTE:


103

1) The SONET/SDH port must be in a shut down state to activate any type of loopback.

2) When you loop back a SONET/SDH port, make sure it is not line timing.

3) The loopback setting is never saved to the generated/saved configuration file.

Task CLI Command

To activate a loopback on the SONET/SDH port

config port <port-id> <enter>

config>port# sonet-sdh loopback {line|internal}

Description:

line — Set the port into line loopback state.

internal — Set the port into internal loopback state.

To disable the loopback on the SONET/SDH port

config>port# sonet-sdh no loopback

TDM ports:

You can use CLI to put a specified TDM port or channel into a loopback mode.

NOTE:

1) The corresponding port or channel must be in a shutdown state in order for the loopback mode to be enabled. The upper level port or channel or parallel channels should not be affected by the loopback mode.

2) When you loop back a port, make sure it is not line timeing.

3) The loopback setting is never saved to the generated/saved configuration file.

Task CLI Command

To activate a loopback on a DS3 port

config port <port-id> <enter>

config>port# tdm ds3 loopback {line|internal|remote}

To disable this specific loopback

config>port# tdm ds3 no loopback

To activate a loopback on a E3 port

config>port# tdm e3 loopback {line|internal|remote}


104


config>port# tdm e3 no loopback

To activate a loopback on a

DS1 channel

config>port# tdm ds1 loopback {line|internal|remote}


config>port# tdm ds1 no loopback

To activate a loopback on a E1 channel

config>port# tdm e1 loopback {line|internal|remote}


config>port# tdm e1 no loopback

6.2. OSPF Problems

This section provides information on how to troubleshoot an OSPF related problem.

6.2.1. Commands common to any OSPF troubleshooting

“show” commands used to check OSPF related configuration

The following commands are commonly used for checking OSPF related configuration:

show router ospf area

show router ospf interface

show router ospf neighbor

show router ospf status

show router ospf database

View the OSPF related alarms/logs

To view the OSPF related alarms or log messages, use the command: show log log-id 99 application ospf

Using “Debug” to troubleshoot an OSPF related problem


105

The debug router ospf command allows the user to troubleshoot an OSPF related issue in many circumstances. The following are the choices of events that can be logged: SR12# debug router ospf - no ospf - ospf [no] area - Enable/disable debugging for an OSPF area [no] area-range - Enable/disable debugging for an OSPF area range [no] cspf - Enable/disable debugging for an OSPF cspf [no] interface - Enable/disable debugging for an OSPF interface [no] leak - Enable/disable debugging for OSPF leaks [no] lsdb - Enable/disable debugging for an OSPF link-state database (LSDB) [no] misc - Enable/disable debugging for miscellaneous OSPF events [no] neighbor - Enable/disable debugging for an OSPF neighbor [no] nssa-range - Enable/disable debugging for an NSSA range [no] packet - Enable/disable debugging for OSPF packets [no] rtm - Enable/disable debugging for OSPF rtm [no] spf - Enable/disable debugging for OSPF spf [no] virtual-neighb* - Enable/disable debugging for an OSPF virtual neighbor

Important Notes:

1) Before enabling “debug”, the user must make sure a log is created to view the debug result. The following is an example log created to view debug results. Refer to 7750_SR_OS_System_Guide_2.0.pdf for more details.

Note that if the log destination is session, when the session is closed, the log (log-id) will not be saved.

For example, log 3 is created to view the debug result:

SR12>config>log>log-id 3 SR12>config>log>log-id$ from debug-trace SR12>config>log>log-id$ to session SR12>config>log>log-id$ no exit

2) To stop the “debug”, use either of the following commands to stop the debug at different levels:

Command Explanation

debug router ospf no packet Disable debugging for OSPF packets

debug router no ospf Disable debugging for all OSPF messages

no debug Disable debugging for all applications

3) The “debug” will stop if a router is rebooted for some reason.


106

6.2.2. OSPF not come up

Symptom: Router OSPF doesn’t come up.

The following table outlines the problems that might cause this symptom and describes suggested actions to resolve the problems.

NOTE: Example outputs of some commands marked with ** are provided after the table.

Possible Problem Suggested Action

To verify if the port is up:

show port

To verify that interface has been assigned a port

show router interface <int-name> detail**

or

config router interface <int-name>

config>router>if# info [detail]**

1. Link/Interface Status

To bind an interface to a physical port, use the command:

config router interface <int-name>

config>router>if# port-id[:encap-val]

Note: encap-val - 0 for null

- [0..4094] for dot1q

2. MTU Mismatch The MTU can be set at the port level or at the IP level. To view the MTU settings, use the following commands:

show port displays MTU at the port level.

show router ospf interface <int-name> detail** displays the IP MTU.


107

Use the commands below to modify MTU setting if it is wrong.

To set the MTU at the port level:

config port <port-id> ethernet mtu <value>

To set the MTU at the IP level:

config router ospf area <area-id> interface <int-name> mtu <value>

To display the interface type , use the command:

show router ospf interface <int-name> detail

Look at the “IF Type” under “State” category.

3. Mismatched Interface Type

To modify the interface type, use the command:

config router ospf area <area-id> interface <int-name> interface-type {broadcast|point-to-point}

4. Mismatched subnet mask or IP address

Check the router and its neighbor’s interface to see if the subnet mask or IP address matches each other. Use the command:

show router interface

To verify if the interface has been configured in OSPF, use the commands:

show router interface to display router interfaces

show router ospf interface to display router interfaces in OSPF

5. Interface not configured in OSPF

To configure an interface in OSPF, use the command:

config router ospf area <area-id> interface <int-name>

6. Router-id not unique Make sure the router has a unique Router ID. Normally a router uses its system interface as its Router ID. A router ID can also be configured specifically. If neither the system interface or router ID are implicitly specified, then the router ID is inherited from the last four bytes of the MAC address.

To view the router-id, use the command:

show router ospf status

To view the system(loopback) interfaces, use the command:

show router interface system


108

To add system interface(loopback) to OSPF, use the command:

config router ospf area <area-id> interface system

If the router’s OSPF neighbor is configured for authentication, the router must be configured to match the authentication. To view the authentication configuration of an interface, use commands:

config router ospf area <area-id> interface <int-name>

config>router>ospf>area>if# info detail**

7. Neighbor is configured for authentication

To configure the authentication on the interface level, use commands:

config router ospf area <area-id> interface <int-name> authentication-type {password|message-digest}

config router ospf area <area-id> interface <int-name> message-digest-key <key #> md5 <md5-key> The following example displays interface authentication configuration command usage: Example: config>router# ospf config>router>ospf$ area 0.0.0.40 config>router>ospf>area# interface “ to-274ferg” config>router>ospf>area>if# authentication-type password config>router>ospf>area>if# authentication-key dilbert config>router>ospf>area>if# no shutdown config>router>ospf>area>if# exit

To view the area of the interface, use the command:

show router ospf interface

8. Incorrect area

To modify the area setting, and configure OSPF on an interface, use:

config router ospf area <area-id>

config>router>ospf>area# interface <int-name>

To display the interval timers setting for an interface, use the command:

show router ospf interface <int-name> detail

9. Mismatched hello/dead interval timers

To modify the interval timers, use the command:

config router ospf area <area-id> interface <int-name> {dead-interval|hello-interval} <value>

Example outputs of some commands marked with **:

1. show router interface <int-name> detail**


109

SR12# show router interface to-rtr22 detail =============================================================================== Interface Table (Router: Base) =============================================================================== ------------------------------------------------------------------------------- Interface ------------------------------------------------------------------------------- If Name : to-rtr22 Admin State : Up Oper State : Up Protocols : OSPF IP Addr/mask : 10.0.1.1/30 Address Type : Primary IGP Inhibit : Disabled Broadcast Address: Host-ones ------------------------------------------------------------------------------- Details ------------------------------------------------------------------------------- If Index : 3 Virt. If Index : 3 Port Id : 1/1/1 If Type : Network Egress Filter: none Ingress Filter : none SNTP B.Cast : False QoS Policy : 1 MAC Address : 8e:51:01:01:00:01 Arp Timeout : 14400 IP MTU : 1504 ICMP Mask Reply : True Cflowd : None ICMP Details Redirects : Number - 100 Time (seconds) - 10 Unreachables : Number - 100 Time (seconds) - 10 TTL Expired : Number - 100 Time (seconds) - 10 ===============================================================================

2. config>router>if# info [detail]** SR12# configure router interface to-rtr22 SR12>config>router>if# info detail ---------------------------------------------- address 10.0.1.1/30 broadcast host-ones port 1/1/1 no arp-timeout no allow-directed-broadcasts icmp mask-reply redirects 100 10 unreachables 100 10 ttl-expired 100 10 exit qos 1 ingress no filter exit egress no filter exit no mac no ntp-broadcast no cflowd no shutdown


110

----------------------------------------------

3. show router ospf interface <int-name> detail** SR12# show router ospf interface to-rtr22 detail =============================================================================== OSPF Interface (Detailed) : to-rtr22 =============================================================================== ------------------------------------------------------------------------------- Configuration ------------------------------------------------------------------------------- IP Address : 10.0.1.1/30 Interface Name : to-sim22 Area Id : 0.0.0.0 Priority : 1 Hello Intrvl : 10 sec Rtr Dead Intrvl : 40 sec Retrans Intrvl : 5 sec Poll Intrvl : 120 sec Metric : 1000 Advert Subnet : True Transit Delay : 1 Auth Type : None Passive : False MTU : 0 ------------------------------------------------------------------------------- State ------------------------------------------------------------------------------- Admin Status : Enabled Oper State : Designated Rtr Designated Rtr : 10.0.1.1 Backup Desig Rtr : 0.0.0.0 IF Type : Broadcast Network Type : Stub Oper MTU : 1504 Last Enabled : 07/27/2004 12:19:27 Nbr Count : 0 If Events : 2 ------------------------------------------------------------------------------- Statistics ------------------------------------------------------------------------------- Tot Rx Packets : 0 Tot Tx Packets : 623 Rx Hellos : 0 Tx Hellos : 623 Rx DBDs : 0 Tx DBDs : 0 Rx LSRs : 0 Tx LSRs : 0 Rx LSUs : 0 Tx LSUs : 0 Rx LS Acks : 0 Tx LS Acks : 0 Retransmits : 0 Discards : 0 Bad Networks : 0 Bad Virt Links : 0 Bad Areas : 0 Bad Dest Addrs : 0 Bad Auth Types : 0 Auth Failures : 0 Bad Neighbors : 0 Bad Pkt Types : 0 Bad Lengths : 0 Bad Hello Int. : 0 Bad Dead Int. : 0 Bad Options : 0 Bad Versions : 0 Bad Checksums : 0 ===============================================================================

4. config>router>ospf>area>if# info detail** (to view the authentication configuration of an interface) SR12# configure router ospf area 0 interface to-rtr20 SR12>config>router>ospf>area>if# info detail ---------------------------------------------- no passive interface-type broadcast priority 1 hello-interval 10 dead-interval 40 retransmit-interval 5 transit-delay 1


111

no mtu no metric no authentication-type no authentication-key no shutdown ----------------------------------------------

6.3. BGP Problems

This section provides information on how to troubleshoot a BGP related problem. Each sub-section describes a possible problem scenario. Examples of command usage are provided in the sub-sections.

6.3.1. Commands common to any BGP troubleshooting

“show” commands used to check BGP related configuration

The following commands are commonly used for checking BGP related configuration:

show router bgp summary

show router bgp neighbor

show router bgp neighbor <ip-address> received-routes

show router bgp neighbor <ip-address> advertised-routes

show router bgp neighbor <ip-address> detail

View the BGP related alarms/logs

To view the BGP related alarms or logs, use the command: show log log-id 99 application bgp

Using “Debug” to troubleshoot a BGP related problem

The debug router bgp command allows the user to troubleshoot a BGP related issue in many circumstances. The following are the choices of events can be logged:

SR12# debug router bgp - bgp - no bgp [no] events - Enable/disable debugging for all BGP events [no] keepalive - Enable/disable debugging for all BGP Keepalive messages [no] notification - Enable/disable debugging for all BGP Notification messages [no] open - Enable/disable debugging for all BGP Open messages [no] packets - Enable/disable debugging for all BGP packets [no] route-refresh - Enable/disable debugging for BGP route-refresh


112

[no] rtm - Enable/disable debugging for addition removal and modification of BGP routes to the system Route Table Manager [no] socket - Enable/disable debugging for all BGP sockets [no] timers - Enable/disable debugging for all BGP timers [no] update - Enable/disable debugging for all BGP Update messages

Important Notes:

1) Before enabling the “debug”, the user must make sure a log is created to view the debug result.

2) To stop the “debug”, use either of the following commands to stop the debug at different level:

Command Explanation

debug router bgp no keepalive Disable debugging for all BGP Keepalive messages

debug router no bgp Disable debugging for all BGP messages

no debug Disable debugging for all applications


6.3.2. BGP peer session not established

Symptom: Router does not establish a session with its peer.

Possible Problem Suggested Action

To verify if the port MTU size is configured correctly, use command: show port <port-id>

1. MTU configuration mismatch

Use config port <port-id> command to modify the MTU size if it is improperly configured. For example, to modify an Ethernet port (1/1/1) MTU size to be 1518 bytes, use command:

config port 1/1/1 ethernet mtu 1518

To verify if the local or Peer AS is configured correctly, use command:


2. Local or Peer AS configured improperly

Use config router bgp command to modify AS number if it is the problem.

For example: to modify the local AS number, use command:


113

config router bgp local-as <as-number>

To modify the (group level) AS number for the remote peer, use command:

config router bgp group <name> peer-as <as-number>

To verify if a BGP neighbor address is configured correctly, use command:


3. BGP neighbor address mis-configured

Use config router bgp group <name> neighbor <ip-address> command to modify the neighbor address if it is incorrect.

Example output of the commands:

1. MTU configuration mismatch

SR12# show port 1/1/1 =============================================================================== Ethernet Interface =============================================================================== Description : 10/100 Ethernet TX Interface : 1/1/1 Speed : 100 mbps Link-level : Ethernet MTU : 1514 Admin state : up Duplex : full Oper state : up Hold time up : 0 seconds Physical Link : Yes Hold time down : 0 seconds IfIndex : 18907136

Last State Change : 07/22/2004 20:14:10 SR12# configure port 1/1/1 ethernet mtu 1518 SR12# show port 1/1/1 =============================================================================== Ethernet Interface =============================================================================== Description : 10/100 Ethernet TX Interface : 1/1/1 Speed : 100 mbps Link-level : Ethernet MTU : 1518 Admin state : up Duplex : full Oper state : up Hold time up : 0 seconds Physical Link : Yes Hold time down : 0 seconds IfIndex : 18907136

Last State Change : 07/22/2004 20:14:10

2. Local or Peer AS configured improperly

SR12# show router bgp neighbor


114

=============================================================================== BGP Neighbor =============================================================================== ------------------------------------------------------------------------------- Peer : 5.5.5.5 Group : iBGP ------------------------------------------------------------------------------- Peer AS : 65531 Peer Address : 5.5.5.5 Peer Port : 179 Local AS : 65531 Local Address : 1.1.1.1 Local Port : 50742 Peer Type : Internal State : Established Last State : OpenSent Last Event : recvKeepAlive Last Error : Hold Timer Expire Local Family : IPv4 Remote Family : IPv4 Local Capability : RouteRefresh MP-BGP Remote Capability: RouteRefresh MP-BGP Hold Time : 90 Keep Alive : 30 SR12>config>router>bgp# info ---------------------------------------------- import "import" export "fromStatic" local-as 65531 router-id 2.2.2.2 group "ibp" exit group "iBGP" type internal peer-as 65531 neighbor 5.5.5.5 exit exit ----------------------------------------------

3. BGP neighbor address misconfigured SR12# show router bgp neighbor =============================================================================== BGP Neighbor =============================================================================== ------------------------------------------------------------------------------- Peer : 5.5.5.5 Group : iBGP ------------------------------------------------------------------------------- Peer AS : 65531 Peer Address : 5.5.5.5 Peer Port : 179 Local AS : 65531 Local Address : 1.1.1.1 Local Port : 50742 Peer Type : Internal State : Established Last State : OpenSent Last Event : recvKeepAlive Last Error : Hold Timer Expire Local Family : IPv4 Remote Family : IPv4 Local Capability : RouteRefresh MP-BGP Remote Capability: RouteRefresh MP-BGP Hold Time : 90 Keep Alive : 30 SR12>config>router>bgp# info ---------------------------------------------- import "import"


115

export "fromStatic" local-as 65531 router-id 2.2.2.2 group "ibp" exit group "iBGP" type internal peer-as 65531 neighbor 5.5.5.5 exit exit ----------------------------------------------

6.3.3. BGP load balancing issue

Route Selection Criteria

When the BGP speaker receives updates from multiple ASs that describe different paths to the same destination, it must choose the single best path for reaching that destination. Once chosen, BGP propagates the best path to its neighbors. The process of selecting the best path is as below.

For each prefix in the routing table, the routing protocol selects the best path. Then, the best path is compared to the next path in list until all paths in the list are exhausted. The following parameters are used to determine the best path: 1. Routes are not considered if they are unreachable. 2. An RTM’s preference is lowered as well as the hierarchy of routes from a different protocol. The lower the preference is, the higher the chance of the route being the active route. 3. Routes with higher local preference have preference. 4. Routes with the shorter AS path have preference. 5. Routes with the lower origin have preference.

IGP = 0 EGP = 1 INCOMPLETE = 2

6. Routes with the lowest MED metric have preference. 7. Routes learned by an EBGP peer rather than those learned from an IBGP peer are preferred. 8. Routes with the lowest IGP cost to the next-hop path attribute are preferred. 9. Routes with the lowest BGP-ID are preferred. 10. Routes with shortest cluster list are preferred. 11. Routes with lowest IP address are preferred.

Commands to adjust BGP attributes for load balancing

Attributes CLI Commands


116

Local Preference Attribute Local preference can be set at the global level:

config>router>bgp local-preference [0..4294967295]

or group level: config>router>bgp>group name local-preference [0..4294967295]

or neighbor level. config>router>bgp>group name>neighbor ip-addr local-preference [0..4294967295]

Note: This command enables setting the BGP local-preference attribute in incoming routes if not specified and configures the default value for the attribute.

This value is used if the BGP route arrives from a BGP peer without the local-preference integer set.

The specified value can be overridden by any value set via a route policy. This configuration parameter can be set at three levels: global level (applies to all peers), group level (applies to all peers in peer-group) or neighbor level (only applies to specified peer). The most specific value is used.

as-path-ignore config router bgp as-path-ignore

This command determines whether the AS path is used to determine the best BGP route.

If this option is present, the AS paths of incoming routes are not used in the route selection process.

MED Attribute MED value can be set at the global level: config>router>bgp med-out {number | igp-cost}

or group level: config>router>bgp>group name med-out {number | igp-cost}

or neighbor level: config>router>bgp>group name>neighbor ip-addr med-out {number | igp-cost}

number — The MED path attribute value expressed as a decimal integer. Values 0 - 4294967295 (2^32 - 1) igp-cost — The MED is set to the IGP cost of the given IP prefix.

This command enables advertising the Multi-Exit Discriminator (MED)


117

and assigns the value used for the path attribute for the MED advertised to BGP peers if the MED is not already set.

The specified value can be overridden by any value set via a route policy.

This configuration parameter can be set at three levels: global level (applies to all peers), group level (applies to all peers in peer-group) or neighbor level (only applies to specified peer). The most specific value is used.

always-compare-med config router bgp always-compare-med {zero | infinity}

Note: This command specifies how the Multi-Exit Discriminator (MED) path attribute is used in the BGP route selection process. The MED attribute is always used in the route selection process regardless of the peer AS that advertised the route. This parameter determines what MED value is inserted in the RIB-IN.

If this parameter is not configured, the router only compares MEDs for routes from external neighbors that are in the same AS.

6.4. Prefix-list (Access-list) in the Route Policy

This section describes with an example how prefix lists (aka. access lists) are configured and used in route policies. “Show” commands are also provided to troubleshooting a route policy related issue.

Overview of the route policy Route policies allow you to configure routing according to specifically defined policies. You can create policies and entries to allow or deny paths based on various parameters such as destination address, protocol, packet size, and community list. Policies can be as simple or complex as required. A simple policy can block routes for a specific location or IP address. More complex policies can be configured using numerous policy statement entries containing matching conditions to specify whether to accept or reject the route, control how a series of policies are evaluated, and manipulate the characteristics associated with a route. There are no default route policies. Each policy must be created explicitly and applied to a policy, a routing protocol, or to the forwarding table. Policy parameters are modifiable.

Process of provisioning a basic router policy

The following diagram shows the process of how to provision a basic route policy. For more detailed description on route policy concept and configuration guidance, please refer to 7750_SR_OS_Router_Guide_2.0.pdf.


118

The following example is focused on how prefix lists are configured and used in a route policy, and how this route policy applied to BGP. Other parameters such as AS-path, community list and damping parameters are disregarded.

1) create/edit route policy

SR12>config>router>policy-options# SR12>config>router>policy-options# begin

2) create/edit prefix lists

SR12>config>router>policy-options# prefix-list “Deny-routes” SR12>config>router>policy-options>prefix-list# prefix 0.0.0.0/8 longer . . . SR12>config>router>policy-options>prefix-list# exit SR12>config>router>policy-options# prefix-list "permit-routes" SR12>config>router>policy-options>prefix-list$ prefix 10.10.1.0/30 exact SR12>config>router>policy-options>prefix-list$ prefix 10.10.2.0/24 . . . SR12>config>router>policy-options>prefix-list$ exit

3) create/edit route policies

SR12>config>router>policy-options# policy-statement "Service Provider-IN" SR12>config>router>policy-options>policy-statement$ entry 1 SR12>config>router>policy-options>policy-statement>entry$ from prefix-list "D eny-routes" SR12>config>router>policy-options>policy-statement>entry# exit SR12>config>router>policy-options>policy-statement>entry# action reject SR12>config>router>policy-options>policy-statement>entry# exit


119

SR12>config>router>policy-options>policy-statement# default-action accept SR12>config>router>policy-options>policy-statement>default-action# exit SR12>config>router>policy-options>policy-statement# exit SR12>config>router>policy-options# policy-statement "Service Provider-OUT" SR12>config>router>policy-options>policy-statement$ entry 20 SR12>config>router>policy-options>policy-statement>entry$ from SR12>config>router>policy-options>policy-statement>entry>from$ prefix-list "p ermit-routes" SR12>config>router>policy-options>policy-statement>entry>from$ exit SR12>config>router>policy-options>policy-statement>entry# action accept SR12>config>router>policy-options>policy-statement>entry>action# exit SR12>config>router>policy-options>policy-statement>entry# exit SR12>config>router>policy-options>policy-statement# default-action reject SR12>config>router>policy-options>policy-statement# exit

4) save route policies

SR12>config>router>policy-options# commit SR12>config>router>policy-options# exit SR12#

5) Apply route policies created above as the import & export policy for BGP

SR12# config router SR12>config>router# autonomous-system <as-number> SR12>config>router# bgp SR12>config>router# import "Service Provider-IN" SR12>config>router# export "Service Provider-OUT" SR12>config>router# exit SR12#

Notes of “begin” and “commit” in the policy configuration:

“begin”

� Required in order to enter the mode to create or edit route policies.

� The ‘begin’ command puts the node (not just the session) in a route policy edit mode.

� Once ‘begin’ is entered, until a commit is executed, subsequent users executing the ‘begin’ command will be warned that a policy configuration is in progress.

“commit”

� This command is required to save changes made to a route policy.

� A ‘commit’ will save all policy configuration in progress on a node, this include all session that have entered ‘begin’ without having exited with a ‘commit’ regardless of the state of the route-policy under configuration.

� A ‘commit’ terminates edit mode for all users that are currently in edit mode.

Troubleshooting Route Policies


120

To verify how the policy is configured, use command: show router policy

To verify how prefix list is configured in the policy, use command: show router policy prefix-list <name>

The following are example outputs of these commands: SR12# show router policy =============================================================================== Route Policies =============================================================================== Policy Description ------------------------------------------------------------------------------- Service Provider-IN Service Provider-OUT ------------------------------------------------------------------------------- Policies : 2 =============================================================================== SR12# SR12# show router policy prefix-list ================================== Prefix Lists ================================== Prefix List Name ---------------------------------- Deny-routes permit-routes ================================== SR12# show router policy prefix-list Deny-routes

prefix 0.0.0.0/8 longer . . .

SR12# show router policy prefix-list permit-routes prefix 10.10.1.0/30 exact

prefix 10.10.2.0/24 exact . . .

SR12#

6.5. Black holing Problems

When an AS provides transit service to other ASs and if there are non-BGP transit routers in the AS, transit traffic might be dropped if the intermediate non-BGP routers haven’t learned the routes for that traffic via IGP. In this case, the transit traffic is black-holed.

By default, Alcatel 7750SR will not re-advertise learned iBGP routes unless there is an entry in its routing table learned via an IGP or a static route.

If you believe that you are black holing a route, you can:

1. Check if the route is in the RIB. Use command show router bgp neighbor <ip addr>

{advertised-routes|received-routes} and show route route-table


121

2. Check if the route is in the FIB. Use command show router fib <slot-number> [<ip-prefix/mask]> [longer]]

3. Verify the routing policies for inaccuracies to ensure that packets are not getting filtered.

- To check what policy is applied in IGP (ex. OSPF), use commands:

config router ospf

config>router>ospf# info detail

- To check if the policy is configured correctly, use command:

show router policy <policy-name>

6.6. LDP not established

This section describes how to troubleshoot problems establishing a LDP.

First make sure the router’s OSPF adjacencies are up and running. If there is anything wrong with OSPF, refer to Section 6.2 for troubleshooting OSPF problems.

If it is not OSPF issue, use the following methods to identify problems in LDP.

View the log messages about LDP

show log log-id 99 subject LDP

Using “Debug” to troubleshoot a LDP related problem

The debug router ldp command allows the user to troubleshoot a LDP related issue. The following are the debugging choices.

SR12# debug router ldp - ldp - no ldp [no] interface + Enable/disable and configure debugging for an LDP interface [no] peer + Enable/disable and configure debugging for an LDP peer SR12>debug>router>ldp# interface <interface-name> [no] event + Configure debugging for specific LDP events [no] packet + Enable/disable debugging for specific LDP packets SR12>debug>router>ldp# peer <ip-address> [no] event + Configure debugging for specific LDP events [no] packet + Enable/disable debugging for specific LDP packets


122

Important Notes:

1) Before enabling the “debug”, the user must make sure a log is created to view the debug result. 2) To stop the “debug”, use either of the following commands to stop the debug at different level (more choices can be found by clicking “?” at any level of the CLI syntax):

Command Explanation

debug router ldp interface <int-name> no packet

Disables debugging for specific LDP packets

debug router ldp no interface <int-name>

Disables debugging for LDP interface

no debug Disables debugging for all applications


Using “show” commands to check LDP information

Command Explanation show router ldp bindings

To display LDP bindings information show router ldp discovery

To display LDP discovery information show router ldp interface

To display LDP interface information show router ldp parameters

To display LDP configured and operation parameters show router ldp peer

To display LDP targeted peer information show router ldp session

To display LDP session information show router ldp status

To display LDP operational information

6.7. CPU Utilization high Scenario

This section provides some possible reasons that could cause high CPU Utilization (e.g. 100%).

To verify the utilization and identify which process is loading the CPU, use the show system cpu command.

Possible reasons why the CPU could be at or near 100%:

• Security issues that cause packets to reach the CPU.


123

o You could create a management filter and logs that could help identify which excessive or unwanted packets are reaching the 7750 SR and block such traffic by modifying the management filter or by using mac/ip filtering.

• Excessive debugging.

o show debug commands will identify the debugging processes running on the 7750. The no debug command is a quick method to stop all debugging.

• Functions such as SNMP MIB walks and large routing updates can cause the CPU to spike to 100%, but in general these functions are temporary and generally have no lasting affect on the performance of the 7750 SR.

6.8. Troubleshooting IES (Internet Enhanced Service) services

This section describes how to troubleshoot an IES service if it is operationally down.

Troubleshooting IP routing protocols:

Before any service is provisioned, the corresponding IP routing protocols must have been configured and running. The IES service could be down if it is related to a routing problem. Refer to other sub-sections in Section 6 for troubleshooting a routing problem.

Verifying IES service configuration

The following table outlines where and how to verify an IES service configuration.

Task CLI Command

To view the configurations related the IES service

Use either of the following commands to view the IES service configuration in different level:

show service service-using

show service id <service-id> all

show service id <service-id> base

show service id <service-id> sap

show service id <service-id> interface

show service id <service-id> arp

To view the port status related to the SAP

If a port/channel is administratively shutdown, all SAPs on that port/channel will be operationally out of service.

show port <port-id>

To view the SAP encapsulation

show service id <service-id> sap <slot/mda/port>**


124

To view the IP interface status

show service id <service-id> interface [<ip-address|ip-int-name>] [detail]**

To delete a SAP on an interface

When a SAP is deleted, all configuration parameters for the SAP will also be deleted. For IES service, the IP interface must be shutdown before the SAP on that interface may be removed.

config service ies <service-id> interface <ip-int-name>

config>service>ies>if# shutdown

config>service>ies>if# no sap <sap-id>

To view IP filter policy (Filter-ID) related to a SAP

Only IP Filter Policies can be applied to IES services.

show service id <service-id> sap <slot/mda/port>**

Look at “Ingress Filter-Id” and “Egress Filter-Id”

or

config service ies <service-id> interface <ip-int-name>

config>service>ies>if# sap <sap-id>

info detail

To view the IP filter policy if Filter-ID is known

show filter ip <ip-filter-id>

1. show service id <service-id> sap <slot/mda/port>** SR12>show>service>id# sap 1/1/4 ============================================================================= Service Access Points(SAP) ============================================================================= Service Id : 100 SAP : 1/1/4 Encap : null Dot1Q Ethertype : 0x8100 QinQ Ethertype : 0x8100 Description : (Not Specified) Split Horizon Group : (Not Specified) Admin State : Up Oper state : Down Last Changed : 07/27/2004 16:07:55 Admin MTU : 1514 Oper MTU : 1514 Ingress qos-policy : 1 Egress qos-policy : 1 Ingress Filter-Id : n/a Egress Filter-Id : n/a Multi Svc Site : None I. Sched Pol : (Not Specified) E. Sched Pol : (Not Specified)


125

Acct. Pol : None Collect Stats : Disabled =============================================================================

2. show service id <service-id> interface [<ip-address|ip-int-name>] [detail]** SR12# show service id 100 interface to-web ============================================================================== Interface Table ============================================================================== Interface-Name Type IP-Address Adm Opr Type ------------------------------------------------------------------------------ to-web Pri 10.3.3.3/24 Up Down IES ------------------------------------------------------------------------------ Interfaces : 1 ============================================================================== SR12# show service id 100 interface to-web detail =============================================================================== Interface Table =============================================================================== ------------------------------------------------------------------------------- Interface ------------------------------------------------------------------------------- If Name : to-web Admin State : Up Oper State : Down Protocols : None IP Addr/mask : 10.3.3.3/24 Address Type : Primary IGP Inhibit : Disabled Broadcast Address: Host-ones Description : (Not Specified) ------------------------------------------------------------------------------- Details ------------------------------------------------------------------------------- If Index : 5 Virt. If Index : 5 Port Id : 1/1/4 If Type : IES SNTP B.Cast : False MAC Address : 8e:51:01:01:00:04 Arp Timeout : 14400 IP MTU : 1500 ICMP Mask Reply : True Cflowd : None ICMP Details Redirects : Number - 100 Time (seconds) - 10 Unreachables : Number - 100 Time (seconds) - 10 TTL Expired : Number - 100 Time (seconds) - 10 ------------------------------------------------------------------------------- Interfaces : 1 ===============================================================================

6.9. Network Monitoring

There are two major ways to monitor the 7750 SR network to detect if there is any trouble – one is through monitoring the event log messages generated on each 7750 SR, the other way is through Alcatel 5620 SAM which is a network level manager that provides fault management functionality.


126

Event logs are the means of recording system generated events for later analysis. Should there exist a fault within a 7750 SR system, event logs are often the first source of information in the troubleshooting process. Events are messages generated by the system for applications or processes within the 7750 SR.

Logs can be configured to collect log messages related to a specific item. When a new log is created, it can be sent to one of the log destinations: Console, Session, Memory log, a Log file, SNMP trap group or Syslog. The operators can then monitor the logs from there. The default log log-id 99 is a memory log and contains all main events. The following is an example of how to create a log and send it to a session.

SR12# configure log log-id 3 SR12>config>log>log-id$ from main SR12>config>log>log-id$ to session SR12>config>log>log-id$ exit

Note that if the log destination is session, when the session is closed, the log (log-id) will not be saved.

For more details of configuring a log, you can also refer to 7750_SR_OS_System_Guide_2.0.pdf.

The 5620 SAM converts SNMP traps from 7750 SR routers to events and alarms. These are then correlated against the managed equipment and configured services and policies. Alarms are applied against the appropriate equipment and services. From the GUI, operators have a number of tools to fine-tune, define, and track alarms. They can:

• View the relationship between incoming alarms and the affected objects, such as the effect of equipment alarms on service operation

• Determine and then set specific policies for each alarm type, for example, the alarm’s incoming severity and its escalated severity

• Track the most important alarms using color codes, for example, sort all red critical alarms.

Figure 17 shows the alarm relationships and the GUI tools to manage them.


127

Figure 17: Alarm relationships on the 5620 SAM GUI

For more information on 5620 SAM fault management features, please refer to “Alcatel 5620 SAM (Release 2.0) General Information Book”.


128

7. Miscellaneous

Commonly Used Global CLI commands

The following is a list of the more commonly used global commands, which means these commands can be executed at any level of the CLI hierarchy.

Global CLI Commands Description

help

?

Displays help in the CLI

history Displays a list of most recently entered commands

info Displays the running configuration for a configuration context

ping verify the reachability of a remote host

pwc Displays the present working context of the CLI session

traceroute Determines the route to a destination addess

tree Displays a list of all commands at the current level and all sublevels


129

History

Version Date Author Reason

0.1 June 09,2004 Stephen Rowlandson,

Cynthia Zhao

2.0 Aug. 05, 2004 Cynthia Zhao

Claude Boulerice

1. Adding more in Section 4.5, 4.6 and 5.2.

2. Adding Section 6 - more troubleshooting notes for commonly occur scenarios.


130

This document contains confidential information which is proprietary to Alcatel. No part of its contents may be used, copied, disclosed or conveyed to any party in any manner whatsoever without prior written permission from Alcatel. Alcatel, the Alcatel logo and all 7750 SR products are registered trademarks of Alcatel.

© Copyright 2004, Alcatel. All Rights Reserved.