rnc and nodeb health check

RNC & NodeB Health Check for NEV 4.2- How To

Document number: UMT/UMT/APP/015074 Document issue: 03.01 / EN Document status: Standard Date: 19/May/2006

Confidential document - Not to be circulated outside Nortel

Copyright 2006 Nortel Networks, All Rights Reserved

Printed in France

NORTEL CONFIDENTIAL

The information contained in this document is the property of Nortel Networks. Except as specifically authorized in writing by Nortel Networks, the holder of this document shall keep the information contained herein confidential and shall protect same in whole or in part from disclosure and dissemination to third parties and use same for evaluation, operation and maintenance purposes only.

The content of this document is provided for information purposes only and is subject to modification. It does not constitute any representation or warranty from Nortel Networks as to the content or accuracy of the information contained herein, including but not limited to the suitability and performances of the product or its intended application.

This is the Way. This is Nortel, Nortel, the Nortel logo, and the Globemark are trademarks of Nortel Networks. All other trademarks are the property of their owners.


Nortel confidential

UMT/UMT/APP/015074 03.01 / EN Standard 19/May/2006 Page 2/300

PUBLICATION HISTORY

19/MAY/2006

Issue 03.01 / EN, STANDARD.

Update of the Document Issue 02.02 to adapt it to the V4.2 version

New HSDPA, ALCAP and RANAP Chapters

OAM Basic checks added


Nortel confidential


CONTENTS 1 INTRODUCTION.......................................................................................................... 8 2 RNC1500 OVERVIEW ................................................................................................. 9

2.1. Hardware changes.................................................................................................... 9 2.2. OMU and TMU: new PMC (ap) roles, with deterministic assignments.................. 9 2.3. RNCIN object evolution oam OMU addresses................................................... 12 2.4. Central OMU Troubleshooting Tools (COTT) (RNC1500-RNC1000) ................ 14

2.4.1 COTT for RNC1000 ........................................................................................ 14 2.4.1.1 How to launch the tool.............................................................................. 14 2.4.1.2 Main Menu................................................................................................ 14 2.4.1.3 Mib Patch tool sub menu .......................................................................... 15 2.4.1.4 Sm_launcher sub menu ............................................................................. 15 2.4.1.5 Snapshot sub menu ................................................................................... 16

2.4.2 COTT for RNC1500 ........................................................................................ 16 2.4.2.1 How to launch the tool.............................................................................. 16 2.4.2.2 Menus........................................................................................................ 17

2.5. Connection to OMU and TMU.............................................................................. 17 2.5.1 Cott tool ........................................................................................................... 17 2.5.2 Telci connection from the passport:................................................................. 18

2.6. CollectCRLog Tool for RNC1500......................................................................... 19 2.7. File System overview for RNC1500...................................................................... 21

2.7.1 New partitions.................................................................................................. 21 2.7.2 Partitions use................................................................................................... 22 2.7.3 OMU local-share directories............................................................................ 22 2.7.4 Navigation on the File System:........................................................................ 23

2.8. How to retrieve OMU and TMU sessions ............................................................. 25 2.9. MIB directories for RNC1500 ............................................................................... 27

2.9.1. Overview......................................................................................................... 27 2.9.2. MIB proto........................................................................................................ 28 2.9.3. MIB Patch file................................................................................................. 28

3 CONTROL NODE OPERATIONAL STATE.......................................................... 29 3.1. Find out if an OMU is active ................................................................................. 29 3.2. Identify the boards in the CN................................................................................. 30 3.3. Check that the boards software is up to date ........................................................ 31 3.4. Read the manufacturer and applicative data on a Control Node board ................. 32 3.5. Check the Control Node led status......................................................................... 34 3.6. Led status from the Control Node consultation ..................................................... 36 3.7. Check if the boards are started, and PEC codes..................................................... 38 3.8. Get OMU/TMU FT processes status: ftViCpTable ............................................... 39 3.9. Get OMU/TMU/CC1 non FT processes status...................................................... 42 3.10. Find out if spares TMU are present ..................................................................... 45 3.11. OMU-TMU board address according to the Module Absolute Position ............. 45 3.12. File system check................................................................................................. 50

3.12.1. OMU disks status.......................................................................................... 51


Nortel confidential


3.12.2. Shared MMS mirroring check up ................................................................. 52 3.12.3. MMS integrity check (number of bad sectors) ............................................. 53

3.13. MIB Verification.................................................................................................. 54 3.13.1. Confirmation that the MIB is built................................................................ 54 3.13.2. Mib customization ........................................................................................ 57

3.14. RNC1000 RNC1500 : telci commands changed .............................................. 59 3.15 ATM check-up on CC1 and TM boards ............................................................... 59

3.15.1. S/UNI state.................................................................................................... 60 3.15.2. ATM planes state .......................................................................................... 60

3.15.2.1. CC1 verifications ................................................................................... 60 3.15.2.2. Verifications of the plane on TM (OMU/TMU) boards ....................... 61

4 INTERFACE NODE OPERATIONAL STATE ...................................................... 63 4.1. Control and Function processors LEDs ................................................................. 63 4.2. Ensure I-Node startup is successful ....................................................................... 63 4.3. File System check up for RNC1500 ...................................................................... 64 4.4. Determine if Base OS has completed initialization ............................................... 64 4.5. Determine if the applications have completed initialization.................................. 65 4.6. Determine if ATM connectivity is available ......................................................... 66 4.7. Determine if the AAL2 ATM Interface is available. ............................................. 69 4.8. Determine if the atmmpe layer is available. .......................................................... 71 4.9. Check PMC-M fully redundant ............................................................................. 71

4.9.1 How to find the PMC Master and serving ....................................................... 71 4.9.2 Connection to a PMC....................................................................................... 73 4.9.3 CID use redundancy......................................................................................... 73

5 NODEB ......................................................................................................................... 78 5.1. How to connect and navigate on a NodeB............................................................. 78 5.2. HW, SW and radio check....................................................................................... 79 5.3. Check that the BTS is radiating and the common channels SCCPCH (PCH, FACH) are up and running. ....................................................................................................... 81 5.4. Iub status ................................................................................................................ 82 5.5. iBTS alarm history check....................................................................................... 82 5.6. Activ FFS............................................................................................................... 83 5.7. iBTS active SW version......................................................................................... 84 5.8. iBTS built? Configuration file ............................................................................... 85 5.9. iBTS Commissionning Data Check ....................................................................... 86 5.10. iBTS Config Data (DLU file) check.................................................................... 88 5.11. To retrieve VCC for OAM, NCP, CCP , DS and NDS ....................................... 93 5.12. CS & PS Call check ............................................................................................. 93 5.13. Connection to other boards .................................................................................. 96 5.14. PCM & SYNCHRO check................................................................................... 96

6 OAM BASIC CHECKS............................................................................................. 102 6.1 Check status of all groups ..................................................................................... 102 6.2 Check status of specific group .............................................................................. 102 6.3 Restart one application in the group ..................................................................... 103 6.4 Restart one group .................................................................................................. 103


Nortel confidential


6.5 Restart all groups .................................................................................................. 104 6.6 SMC alarms check ................................................................................................ 105

7 HOW TO................................................................................................................. 108 7.1. How To Wicl.................................................................................................... 108 7.2. How To retrieve the IP address of the nodes ................................................... 110

7.2.1. Control Node Addresses ............................................................................... 110 7.2.2. Interface Node addresses .............................................................................. 110

7.3. CallTrace AccessSession creation for an IMSI through WICL (OMC V5.0) ..... 110 7.3.1. Creation at the OMC..................................................................................... 111 7.3.2. Calltrace creation check on the RNC............................................................ 113 7.3.3. Deletion at the OMC..................................................................................... 115 7.3.4. Calltrace storage............................................................................................ 116

7.4. To retrieve NodeB and Cell information ............................................................. 117 7.5. OAM-RNC link check up. ................................................................................... 124

7.5.1. Control from the Control Node..................................................................... 124 7.5.2. Control from the IN / AN.............................................................................. 125 7.5.3. Verification on the OMC side....................................................................... 125

7.6. Consistency of the SW between RNC and OAM ................................................ 127 7.6.1 RNC Delivery Naming .................................................................................. 127

7.6.1.1. Versions and Deliveries correspondence ............................................... 127 7.6.1.2. RNC delivery naming convention ......................................................... 128 7.6.1.3. Finding the SRS descriptions on the OMC............................................ 129

7.6.2 OMC Versions ............................................................................................... 130 7.6.2.1. Control Node SW from the OMC point of view.................................... 130 7.6.2.2. Interface Node SW from the OMC point of view.................................. 131

7.6.3 CN / IN versions ........................................................................................... 133 7.6.3.1. Control Node SW - OMU check............................................................ 133 7.6.3.2. Interface Node SW - CP check .............................................................. 133

7.7. Check CN-IN link ................................................................................................ 135 7.8. Call check............................................................................................................. 153

7.8.1. RDT Ressource Debug Toolkit..................................................................... 153 7.8.2. TMU.............................................................................................................. 157 7.8.3. RBSet & AsConf Id ...................................................................................... 170 7.8.4. Establishment cause (REC 25.331 R99)....................................................... 172

7.9. IN Counters on User Plane .................................................................................. 173 7.10. IU/IUR links check ............................................................................................ 179 7.11. OMU and TMU VCC info................................................................................. 189 7.12. How To debug SS7 on the RNC-Interface Node........................................... 190 7.13. Observations storage.......................................................................................... 190 7.14. HSDPA How To ........................................................................................... 195

7.14.1. Principle ...................................................................................................... 195 7.14.2. How to .. Check configuration.................................................................... 195 7.14.2.1 Basic configuration ................................................................................... 195 7.14.3. How to .. Check and Troubleshoot HSDPA services on the RNC ............. 200 7.14.4. How to .. Check and Troubleshoot HSDPA services on the BTS .............. 209


Nortel confidential


7.15. ALCAP/RANAP How To ............................................................................. 220 7.15.1. How to .. Check and Troubleshoot ALCAP services ................................. 220 7.15.2. How to .. Check and Troubleshoot RANAP services................................. 227

8 TOOLS........................................................................................................................ 231 8.1. UMTS Log Collection Engine ............................................................................. 231

8.1.1. Documentation/tool....................................................................................... 231 8.1.2. Purpose.......................................................................................................... 231 8.1.3. Quick start..................................................................................................... 231 8.1.4. Screenshots of OAM V4.x............................................................................ 232 8.1.5. Screenshots For OAM V5.x.......................................................................... 233

8.2. RNC traffic check and cell diagnose tool ............................................................ 235 8.2.1. Documentation.............................................................................................. 235 8.2.2. Tools and logs location .............................................................................. 235 8.2.3. Overview of RNC Traffic Check.................................................................. 235

8.2.3.1 Purpose.................................................................................................... 235 8.2.3.2 How to launch the tool Example.......................................................... 236

8.2.4. Overview of Cell Diagnose........................................................................... 240 8.2.4.1. Purpose................................................................................................... 240 8.2.4.2 How to launch the tool Example.......................................................... 240

8.3. WinRNCSpy (IpSpy, AtmSpy)............................................................................ 249 8.3.1. Documentation/Tool ..................................................................................... 249 8.3.2. AtmSpy overview ......................................................................................... 249

8.3.2.1. Tool presentation ................................................................................... 249 8.3.2.2. Passport configuration ........................................................................... 250 8.3.2.3. Definition of the Vcc Capture settings................................................... 250 8.3.2.4. Capture mode ......................................................................................... 251 8.3.2.5. Ftp mode ................................................................................................ 254 8.3.2.6. Convert mode (optional) ........................................................................ 255 8.3.2.7. Decode mode ......................................................................................... 256 8.3.2.8. Manual launch........................................................................................ 257

8.3.3.. Overview of IPSpy....................................................................................... 258 8.3.3.1. Usage...................................................................................................... 258 8.3.3.2. Manual commands ................................................................................. 265

8.4. CTBL ................................................................................................................... 266 8.4.1 CTBL traces through CallTrace wizard......................................................... 266 8.4.2. Manual CTBL traces on a cell. ..................................................................... 266

8.4.2.1. How to retrieve a cell tracekey .............................................................. 266 8.4.2.2. CTBL traces setup on a cell. .................................................................. 269 8.4.2.3. Traces collection .................................................................................... 270 8.4.2.4. Traces decoding ..................................................................................... 270

8.5. Cott....................................................................................................................... 271 9 ANNEXE A : RNC 1500 COMMANDS .................................................................. 272


Nortel confidential



1 INTRODUCTION OBJECT/ SCOPE OF THIS DOCUMENT Basic health checks on RNC1000/ RNC1500/ iBTS /OAM equipments. Applicable to versions UA V4.2 AUDIENCE FOR THIS DOCUMENT GPS, TAS, ER and other operational teams CONVENTIONS The commands to type are in bold and blue. The results to take care of are in red. The commands preceded by:

CLASSIC

apply for RNC1000. The commands preceded by:

COMPACT

apply for RNC1500. As the commands only may differ, refer to the classic part for more detailed explainations of the expected results. Commands impact:

Control Node commands are unsafe on TMU and ci_mgr, they can be very impacting on a loaded RNC if misused.

Passports displays are safe but connections on PMC for unsafe commands have to be carefully used.

iBTS commands provided are not disruptive (except for III 8.)

Nortel confidential



Nortel confidential


2 RNC1500 OVERVIEW Purpose: short presentation of the RNC1500.

2.1. Hardware changes The RNC 1500 consists only in one shelf. The INode shelf will now concentrate both INode and CNode applications. Modifications performed: 9 Removal of the C-Node HW (CC1, TMU-R, OMU, MMS) from the RNC

platform. 9 Porting of TMU s/w to the PSFP 9 Porting of OMU s/w to the PSFP, with some s/w adaptation due to CP3 Disk. 9 Use of a single 20 Gbyte disk for file management

A previous RNC1000 dual-shelf may contain now two PP 15K shelves (i.e. two RNC1500).

2.2. OMU and TMU: new PMC (ap) roles, with deterministic assignments. The OMU and TMU are now only logical entities spread over the PMC. PMC-M (master), RAB (radio access bearer), PC (protocol converter) and NI (in charge of ss7 stacks) already running on the PMC are now completed by OMU and TMU AP. Below an example truncated:


11> d lp/* ap/* rnc processorRole Lp/* Ap/* RncAp +==+==+-------+---------------------------------------- |Lp|Ap| role |Response +==+==+-------+---------------------------------------- | 2| 0|aMaste | | 2| 1|tmu | | 2| 2|rab | | 2| 3|rab | | 2| 4|pc | | 2| 5|rab | | 3| 0|sMaste | | 3| 1|tmu | | 3| 2|rab | | 3| 3|rab | | 3| 4|pc | | 3| 5|rab | | 4| 0|rab | | 4| 1|tmu | | 4| 2|aNi | | 4| 3|rab | | 4| 4|pc | | 4| 5|aOmu | | 5| 0|rab | | 5| 1|tmu | | 5| 2|sNi | | 5| 3|rab | | 5| 4|pc | | 5| 5|sOmu | ok 2006-05-11 13:51:42.42

Nortel confidential



The repartition of OMU/TMU across the PSFP is deterministic. The same pattern is always used, according to the number of PSFP.

AP role and repartitions according to the number of PSFP

1 PSFP 1 PSFP AP0 AP1 AP2 AP3 AP4 AP5 Slot 2 (10)

Nortel confidential


PMCM TMU NI RAB PC OMU

2 PSFP 2 PSFP AP0 AP1 AP2 AP3 AP4 AP5 Slot 2 (10) PMCM TMU NI RAB PC OMU Slot 3 (11) PMCM TMU NI RAB PC OMU

3 PSFP 3 PSFP AP0 AP1 AP2 AP3 AP4 AP5 Slot 2 (10) PMCM TMU RAB RAB PC RAB Slot 3 (11) PMCM TMU NI RAB PC OMU Slot 4 (12) RAB TMU NI RAB PC OMU

From 4 to 12 PSFP

slot Card type 0 1 2 3 4 50 CP1 CP2 PSFP PMC-M TMU RAB RAB PC RAB3 PSFP PMC-M TMU RAB RAB PC RAB4 PSFP RAB TMU NI RAB PC OMU5 PSFP RAB TMU NI RAB PC OMU6 PSFP RAB TMU RAB RAB PC RAB7 PSFP RAB TMU RAB RAB PC RAB8 OC39 OC310 PSFP RAB TMU RAB RAB PC RAB11 PSFP RAB TMU RAB RAB PC RAB12 PSFP RAB TMU RAB RAB PC TMU13 PSFP RAB TMU RAB RAB PC TMU14 PSFP RAB TMU RAB RAB PC RAB15 PSFP RAB TMU RAB RAB PC RAB


2.3. RNCIN object evolution oam OMU addresses d p rncin Perform this command on passport shelf to determine:

the IP addresses provisioned for the OMU the mode of the passport shelf.

The mode must be rnc for an RNC1500 (inode for RNC1000). 21> d -p rncIn RncIn primaryOmuIpAddress = 0.0.0.0 secondaryOmuIpAddress = 0.0.0.0 oamPrimaryOmuIpAddress = 47.164.76.82 OMU EXTERNAL ADDRESS oamSecondaryOmuIpAddress = 47.164.76.83 OMU EXTERNAL ADDRESS oamOmcIpAddresses = 47.164.77.116 OMC SERVER ADDRESS rncInfo = "" protocolConverterResources = 17 % mode = rnc ok 2005-07-20 18:02:13.40 The oamxxxOmuIpAddresses are a mandatory for the link between the RNC and the OMC server. If these addresses are badly defined, no oam operation (such as build, upgrades,) is possible. They must be the same on the RNC and the OMC. To check their definition on the OMC server, use wicl. In our case, the RNC is RNC_1209. saba:/home/nortel (nortel) % /opt/nortel/shell/wicl/wiclsh --------------------------------------------------------- Starting Wireless Internet Command Language --------------------------------------------------------- Command mode launched login : nortel password : nortel ....wait please Wireless Internet Command Language (WICL) >: utran list rnc {RNC RNC333} {RNC RNC00} {RNC RNC_1210} {RNC RNC_1209} {RNC RNC_1202} {RNC RNC_1205} {RNC RNC_1206} {RNC RNC1407} {RNC RNC_1201} >: RNC RNC_1209 list children CallTrace CsCoreNetworkAccess GSMNeighbour LocationServices MtpSignPoint NeighbouringRNC NodeB ObjectTrace PerfScanner PsCoreNetworkAccess RNCEquipment RadioAccessService RncMIB RouteSet SasAccess >: RNC RNC_1209 list RNCEquipment {RNC RNC_1209 RNCEquipment 0}

Nortel confidential



Nortel confidential


>: RNC RNC_1209 RNCEquipment 0 list children ANode CNode INode >: RNC RNC_1209 RNCEquipment 0 list CNode {RNC RNC_1209 RNCEquipment 0 CNode 0} >: RNC RNC_1209 RNCEquipment 0 CNode 0 dis activeIPAddress 47.164.76.82 availabilityStatus { } class CNode dn (RNC,259)(RNCEquipment,0)(CNode,0) ipAddress 47.164.76.82 mibSynchronization on neId 47.164.76.82 operationalState enabled operatorId 0 passiveSystemRelease 04_02_00 secondaryIPAddress 47.164.76.83 siteName Site/SITE_RNC_1209 systemRelease 04_02_00 unknownStatus unknownstatusfalse upgradeStatus IDLE userSpecificInfo NONE You may also ping the OMC from the RNC to check the link RNC OMC. 85> ping -ip(47.164.77.116) vr/0 ip icmp Vr/0 Ip Icmp IpPing: Rx reply of 64 bytes from 47.164.77.116: icmpSeq=2, time=2ms ok 2005-08-03 12:04:29.02 vr/0 was used because the protocol port OAMENET0 is under this virtual router: 83> d vr/* pp/* Vr/* Pp/* Use -noTabular to see hidden attributes: atmAddress. +================+====================+------+-----+----+-----+----- | Vr | Pp |snmpOp|osiAd|osiO|osiUs|ifInd | | |erStat| min |per | age | ex | | | us | | | | +================+====================+------+-----+----+-----+----- |0 |NODEBSOAM0 |up |unlck|ena |activ| 3 |0 |OAMENET0 |up |unlck|ena |activ| 7 [] ok 2005-08-03 12:04:01.61


2.4. Central OMU Troubleshooting Tools (COTT) (RNC1500-RNC1000) COTT is a tool running on the OMU for RNC1000 and on the OMC Main Server for RNC1500. It gathers the following troubleshooting tools and commands:

getCellInfo collectCRLog MIB patch tool Core Process table display IN_Access table display OMU state Messaging full mesh, LWG, routing table List of process launched by SM launcher Access to Telci commands (RNC1500 only) CNode Snapshot (RNC1000 only)

2.4.1 COTT for RNC1000

2.4.1.1 How to launch the tool Open a telnet session on the active OMU (some tools are only available on the active omu) and enter cott.

2.4.1.2 Main Menu =================================================== Central OMU Troubleshooting Tools =================================================== Please select the following tools: 1. collectCRlog 2. IN_Acc_List_All_Tables 3. ftViCpTable 4. getCellInfo 5. mib_patch_tool 6. sm_launcher 7. snapshot 8. msgViLwgTable 9. msgViFullMeshTbl 10. msgViRoutingTable 11. msgViOmuState L. Enable log. Q. Quit menu. Your choice:

Nortel confidential



Note

All the results can be logged on the OMU under the /tmp/COTTlog directory except for collectCRlog (under /tmp)

The IN_Acc_List_All_Tables command result will be logged in the blogio.tmp file.

2.4.1.3 Mib Patch tool sub menu Be careful to type the MIB path before: MIB=/OMU/share/rw_data/MibRunning/DB_TEST.do

Nortel confidential


=================================================== MIB Patch Tool (mib_patch_tool.exe) Note; to use running MIB, type; MIB=/OMU/share/rw_data/MibRunning/DB_TEST.do MIB= (Type MIB=Value to set value) format= (Type format=Value to set value) =================================================== Please select the following options: 1. Read field(s) 2. Write fields(s) 3. Display field(s) type 4. Display field(s) range 5. Display field(s) size 6. Display field(s) offset 7. List the MIB tables 8. Compare MIB with Reference MIB 9. Show patch information stored in the MIB 10. Remove the patch information stored in the MIB. 11. Validate the MIB to check for corruption. H. Help. L. Disable log. Q. Quit menu. Your choice:

2.4.1.4 Sm_launcher sub menu SM Launcher =========== Operating on local board: [Platform: CONTROL NODE; Instance: 0; Sub equip.: SBC; Shelf: 1; Slot: 9] Command Description ------- ----------- create Create a process group modify Modify a process group delete Delete a process group get_create_data Get process group create data ps Process status pgs Process group status


bs Board status board Set board position board-shutdown Board shutdown oam-shutdown OAM shutdown quit Quit Command: Note

In case no board position is previously set, it will automatically display the processes associated with the current board (active OMU)

To display processes on a specific board, set the board position first and choose the ps command

2.4.1.5 Snapshot sub menu =================================================== Emergency Recovery Procedure Tools =================================================== Please select the following procedures: 1. Save the CNode software & the CNode configuration 2. Save the CNode configuration 3. Restore the CNode software and config 4. Restore the CNode config L. Disable log. Q. Quit menu. Your choice:

2.4.2 COTT for RNC1500

2.4.2.1 How to launch the tool COTT has to be launched from the OMC server. The tool is delivered as a part of the RNC load (under the passport cnp software). Each time a new RNC SRS is downloaded on the OMC server, COTT is automatically installed under: /opt/nortel/data/srs/DELIVERY/PASSPORT/software/cnp//external with RI41xxxxxxxxx, the software name of the RNC1500. To launch the tool : ./cott c

Nortel confidential



The login and password are those used for a telnet on the CP. Logging Path: all the results can be logged on the OAM server under the /tmp/COTT directory (example: log Dir: /tmp/COTT/47_164_52_232_20060516T092354). The logging is activated/deactivated by the choice L: Enable log / Disable log.

Nortel confidential


Example for a RI42111061220 load: bernard:/home/nortel (nortel) % cd /opt/nortel/data/srs/DELIVERY/PASSPORT/software/cnp/RI42111061220/external/ bernard:/opt/nortel/data/srs/DELIVERY/PASSPORT/software/cnp/RI42111061220/external (nortel) % ./cott -c 47.164.52.232 Which OMU do you want to logon(0-act/1-sby):0 login name: debug login used to connect onto the CP debug's password: debug ========================================================= Central OMU Troubleshooting Tools ========================================================= Please select the following tools: 1. Collect CR Log 2. INACC_CA display_dd_ctx 3. cpTable 4. getCellInfo 5. mib 6. sm launcher 7. msg lwg 8. msg fullmesh 9. INACC_LA inacc_disp_table_ctx 10. msg omu L. Enable log T. telnet CI Q. Quit menu Your choice:

2.4.2.2 Menus The choices are the same as for RNC1000, except for certain commands naming, slightly different. Note: the getCellInfo command will provide a different output when running from the COTT menu (see chapter 7.4 for details)

2.5. Connection to OMU and TMU

2.5.1 Cott tool The Cott Tool allows a direct connection on the active OMU (T. telnet CI)


2.5.2 Telci connection from the passport: Towards a RAB: telci rab(instance) rnc instance=x Towards a PMC-NI: telci pmcni(instance) rnc instance=0,1 Towards a PMC: telci pmcm(instance) rnc instance=0,1 Towards a TMU: telci tmu(instance) rnc instance=0.1 Towards an OMU telci omu(instance) rnc instance=0,1 To leave the connection menu, enter Q. To link an instance and its lp ap numbers display the attributes instances and roles of lp ap rnc objects: 3> d lp/* ap/* rnc role,instance Lp/* Ap/* RncAp +==+==+-------+----------+---------------------------------------- |Lp|Ap| role | instance |Response +==+==+-------+----------+---------------------------------------- | 2| 0|aMaste | 0| | 2| 1|tmu | 0| | 2| 2|rab | 0| | 2| 3|rab | 1| | 2| 4|pc | 0| | 2| 5|rab | 2| | 3| 0|sMaste | 1| | 3| 1|tmu | 1| | 3| 2|rab | 3| | 3| 3|rab | 4| | 3| 4|pc | 1| | 3| 5|rab | 5| | 4| 0|rab | 6| | 4| 1|tmu | 2| | 4| 2|aNi | 0| | 4| 3|rab | 7| | 4| 4|pc | 2| | 4| 5|aOmu | 0| | 5| 0|rab | 8| | 5| 1|tmu | 3| | 5| 2|sNi | 1| | 5| 3|rab | 9| | 5| 4|pc | 3| | 5| 5|sOmu | 1| ok 2006-05-11 14:01:21.90 In this example, the active OMU is lp/4 ap/5, with the instance 0. To connect it, use telci and then type enter to display the menus (result truncated below),

Nortel confidential



5> telci -omu(0) rnc Trying 127.1.4.7... Connected to 127.1.4.7. Escape character is '^]' Telnet CI commands: Q - quit telci G - redirect global output. telci(omu)>

2.6. CollectCRLog Tool for RNC1500 The Collect CR Log tool is the first option offered by COTT. The files collected are tarred and zipped in a single archive on the OMC server under the directory

/tmp/COTT Example for a collect beginning the 10th of May 2006: ========================================================= Central OMU Troubleshooting Tools ========================================================= Please select the following tools: 1. Collect CR Log 2. INACC_CA display_dd_ctx 3. cpTable 4. getCellInfo 5. mib 6. sm launcher 7. msg lwg 8. msg fullmesh 9. INACC_LA inacc_disp_table_ctx 10. msg omu L. Enable log T. telnet CI Q. Quit menu Your choice: 1

Nortel confidential



From which date do you want collect log (YYYYMMDD): 20060510 Please enter command file (optional): __________________________________________________________ _

___________________________________________________________ % COLLECT CR LOG SCRIPT FOR RNC1500 % % Script started. % % Please be patient, as the script might % take several minutes to run to completion. % %__________________________________________________________ Creating temporary directory structure... Getting info from RNC via TELNET session... [] Bundling files... Deleting temporary directory structure... ___________________________________________________________ % SCRIPT RUN TO COMPLETION % % Start time T155722 % End time T161003 % Size before bundling: 476016 Kb % Output directory: /tmp/COTT/47_164_52_232_20060511T155647 % Output filename: collectCRlog_47_164_52_232_20060510.tar.gz %__________________________________________________________ %__________________________________________________________ The following files are retrieved:

Content of the sessions for OMU 0 and 1 stored under: /CP0/OMU0/local/log/bos/sc/ /CP1/OMU0/local/log/bos/sc/ /CP0/OMU1/local/log/bos/sc/ /CP1/OMU1/local/log/bos/sc/

TMU trace files: /CP0/TMU/local/log/bos/sc/ /CP1/TMU/local/log/bos/sc/

Fault files: /sfs2/share/fault/

Histo files /sfs2/share/log/historisation/*SBC.hst

INode alarms: /spooled/closed/alarm/

Nortel confidential



INode Field traces: /spooled/closed/debug/

INode postmortem: /spooled/closed/appl/cdcsLp*/**

The textual results of the following commands run on CP and OMU: o d -p vr /* on CP o d lp/* ap/* r on CP (roles and location of PMCs) o d sw avl on CP (software version) o d sw patchlist on CP (patchlist) o d fs part/* on CP (disk size and usage) o d lp/0 on CP (uptime) o msg omu on OMU0/1 (Is omu simplex/duplex active/passive) o msg fullmesh on OMU0/1 (list all tcp connections with the boards) o cpTable on OMU0/1 (list all the call processes) o os2 routeShow on OMU0/1 (display host and network routing tables)

Textual results of any additional commands specified in a given command file.

2.7. File System overview for RNC1500

2.7.1 New partitions Compared to an Inode for RNC1000, two new partitions are added. 12> list fs part/* Fs Part/0 Fs Part/1 Fs Part/2 ok 2005-08-08 15:28:11.55 13> list fs disk/* Fs Disk/0 Fs Disk/1 ok 2005-08-08 15:28:16.17 11> d fs part/* Fs Part/* +====+----------+----------+-----+------+-----+----+----- |Part| capacity |freeSpace |usage| mode |osiAd|osiO|osiUs | | Gbyte| Gbyte| %| | min |per | age +====+----------+----------+-----+------+-----+----+----- | 0| 4.294770| 1.122697| 73|shadow|unlck|ena |activ | 1| 7.837175| 7.407804| 5|shadow|unlck|ena |activ | 2| 7.837175| 7.386066| 5|notSha|unlck|ena |activ ok 2005-08-08 15:08:21.55 The first disk partition is untouched for backward compatibility.

Nortel confidential



These partitions are spread over the two disks of each CP: 29> d fs disk/* part/* Fs Disk/* Part/* +====+====+----------+----------+----------+-----+----+----- |Disk|Part| capacity |freeSpace |badBlocksP|osiAd|osiO|osiUs | | | Gbyte| Gbyte|ercentage | min |per | age | | | | | %| | | +====+====+----------+----------+----------+-----+----+----- | 0| 0| 4.294770| 1.122697| 0|unlck|ena |activ | 0| 1| 7.837175| 7.408001| 0|unlck|ena |activ | 0| 2| 7.837175| 7.371448| 0|unlck|ena |activ | 1| 0| 4.294770| 1.122697| 0|unlck|ena |activ | 1| 1| 7.837175| 7.407804| 0|unlck|ena |activ | 1| 2| 7.837175| 7.385219| 0|unlck|ena |activ ok 2005-08-08 15:36:32.28

2.7.2 Partitions use Partition Size (GB) Known as Use

Nortel confidential


/sfs (shadowed)

Passport system use download, provisioning, DCS. All the DCS logs like alarms, debug etc are located under /sfs.

0 4

/local Active CPs view of the disk.

/remote Standby CPs view of the disk.

/sfs/sfs2 (shadowed)

New shadowed partition for use by compact. Including logs for calltrace, counters, fault, and historization are located under this directory.

1 8

/local2 Active CPs view of the above.

/remote2 Standby CPs view of the above.

/sfs/CP0 New non-shadowed partition on the CP in slot 0 for Compact use.

2 8

New non-shadowed partition on the CP in slot 1 for Compact use. /sfs/CP1

2.7.3 OMU local-share directories sfs2 is chiefly dedicated to the mib and the files shared by the OMUs (/OMU/share from a classical OMU prospective). The partition 2 is dedicated to the OMU and TMU (previously /OMU/local/) /CPx/OMU0/ /CPx/OMU1/ /CPx/TMU/


Cp view External view OMU0 view OMU1 view

Nortel confidential


/sfs/sfs2/share /sfs2/share /OMU/share /OMU/share CP0

/sfs/CP0/OMU0/local /CP0/OMU0/local /OMU/local ----------------

/sfs/CP0/OMU1/local /CP0/OMU1/local ---------------- /OMU/local

/sfs/sfs2/share /sfs2/share /OMU/share /OMU/share CP1

/sfs/CP1/OMU0/local /CP1/OMU0/local /OMU/local ----------------

/sfs/CP1/OMU1/local /CP1/OMU1/local ---------------- /OMU/local

2.7.4 Navigation on the File System: ls fs lists the files cd path(path) fs to change the directory pwd fs Path of the working dir.

Example: 67> cd -path("/sfs2/share") fs Fs ok 2005-08-05 14:43:56.17 68> ls fs Fs Type Prot Size Date Time Name dir no 0 2001-06-14 02:59:40 . dir no 0 1980-01-01 00:00:00 .. dir no 0 2001-06-14 02:59:40 rw_data dir no 0 2001-09-02 01:23:42 config dir no 0 2001-09-02 01:23:42 fault dir no 0 2001-09-02 01:23:42 log dir no 0 2001-09-02 01:23:44 usr ok 2005-08-05 14:44:00.51 69> pwd fs Fs /share ok 2005-08-05 14:44:03.25 70> cd -path("/") fs Fs ok 2005-08-05 14:48:12.41


71> ls fs Fs Type Prot Size Date Time Name dir no 0 1980-01-01 00:00:00 tmp dir no 0 1980-01-01 00:00:00 bvw dir no 0 1980-01-01 00:00:00 provisioning dir no 0 1980-01-01 00:00:00 system dir no 0 1980-01-01 00:00:00 spooled dir no 0 1980-01-01 00:00:00 software dir no 0 2001-06-14 02:59:40 debugTrace ok 2005-08-05 14:48:15.90 It is much easier to use a client for ftp browsing.

Nortel confidential



2.8. How to retrieve OMU and TMU sessions To retrieve the logs of the active OMU and the TMU

1. Determine the active CP (os pcsShowCards) => CPi with i = 0 or 1 2. Determine the instance of the active omu (d lp/* ap/* rnc role,instance) =>

OMUj j = 0 or 1 3. Open an FTP connection onto the CP, the paths are

/CPi/OMUj/local/log/bos/sc /CPi/TMU/local/log/bos/sc

Example: 10> d lp/* ap/* rnc role,instance Lp/* Ap/* RncAp +==+==+-------+----------+---------------------------------------- |Lp|Ap| role | instance |Response +==+==+-------+----------+---------------------------------------- [] | 5| 5|aOmu | 1| [] 11> os pcsShowCards CARD TYPE FSM STATE VISIBILITY LP INSTANCE ---- ---------- ----------------- ---------- -- -------- 0 CPeE RunningAsStandby OnLine 0 Standby 1 CPeE RunningAsActive OnLine 0 Active [] The instance of the active OMU is 1, the CP 1 is active, the log files for the active OMU and for the TMU must be retrieved under: /CP1/OMU1/local/log/bos/sc /CP1/TMU/local/log/bos/sc The files can be viewed using the Passport Text viewer: How to use it: 1) Log onto CP 2) Enter OS unsafe mode, change to the directory where the file exists and list the file name 3) Launch the tool

Nortel confidential


16> os Debug tracing directed to this OS shell.


WARNING: Commands available at this interface may be service impacting. This interface is for debugging only. Do not proceed without proper authority and training. Do you wish to proceed? [Yes/No]: yes OS> OS> set unsafe WARNING: Entering unsafe mode. Commands available from this interface are not supported for field use without specific direction from Nortel Networks. OS> OS> set promote OS> OS> cd /sfs/CPi/OMUj/local/log/bos/sc OS> pwd /sfs/CP0/OMU1/local/log/bos/sc OS> OS> ls . .. blogOMU1.20060511T170251.ini CmdServer_1.log blogOMU1.01.trc blogOMU1.20060512T015332.tmp OS> tv blogOMU1.20060511T170251.ini ------------------------------------------------------------------------------- 11/05/2006 17:01:22.107 num:00000000 vxw_io_driver.cc.708 [0xff3f000] tCnStarte r @ OMU_1(Lp/5,Ap/5) (PERM): STDOUT/SRDERR traces "cn_start_bos: vxw_io_initialize -> OK :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: C-Node software Startup Info for OMU_1(Lp/5,Ap/5): Cn software Start Time = 2006-05-11 17:01:18.72320000 Cn Started after 17080 ticks (ms) since PMC reset Environment variables used to configure BaseOS and cNode App: (global environment) 0: BTHD_MAX_NUMBER_OF_THREADS=200 1: BTHD_MAX_NUMBER_OF_KEYS=50 2: DYM_MAX_NUMBER_OF_PARTITIONS=40 3: TRANSIENT_MEM_ENGINEERING=3 4: MP_BSEM_MAX_NUMBER_OF_SEMAPHORES=120 5: MTX_MAX_NUMBER_OF_MUTEXES=130 6: BJOB_MAX_NUMBER_OF_QUEUES=10 7: MP_VAR_SEGMENT_SIZE=8192 8: DLY_MAX_NUMBER_OF_DELAYS=5000 9: OSIROOT=/Osiroot 10: OSIPIPE=/Osipipe 11: APPTR_VERSION=2 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: [] . 4) Use 1 mode if required

Nortel confidential



Nortel confidential


View commands: Command Operation Ctrl-f Page Down Ctrl-b Page up k Scroll Up j Scroll down /str Forward Search / Next forward Search ?str Backward Search ? Next backward Search :0 Top of the file G Bottom of the file h Display help information Refresh current screen q Quit Note:

Search command does not support pattern search During tv active session, alarm messages may insert into the tv screen output.

User could use key to refresh the current page. If alarms are too disruptive, user could use the CAS command to restrain alarm by issuing the following command:

To find your telnet session d nmis tel sess/* d nmis local sess/* To restrain alarm for session 2: set nmis tel sess/2 data ~alarm set nmis local sess/2 data ~alarm

2.9. MIB directories for RNC1500

2.9.1. Overview 2 directories are dedicated to the MIB in the new mounting point sfs2: /sfs2/share/rw_data/Mib1 /sfs2/share/rw_data/Mib2 One contains the running MIB, whilst the second is used for the new MIB built online or during upgrades with changes. The file /sfs2/share/rw_data/mibDirLinks.txt assigns the role of each directory:

its first line defines the MibRunning


Nortel confidential


its second line, the MibNew Each MIB directory contains, like for the classic:

DB_TEST.do the MIB itself SAVE_DB_TEST.do backup in case of write failure on DB_TEST.do mibNominal.txt points out the sane MIB file.

2.9.2. MIB proto Directory : /sfs/software/cRNCApc/obj/ppc/ The proto MIB is compressed Example :

/sfs/software/cRNCApc/obj/ppc/DB_TEST_RI41119052630.do.gz

2.9.3. MIB Patch file The file /sfs2/share/rw_data/mibPatch_Rxxxxxxx.txt (with xxx: the software version), is used to patch the MIB. This file is optional and is taken into account only if the version xxxx in the file name is the running version. It automatically patches the MIB.


3 CONTROL NODE OPERATIONAL STATE Purpose: check that omu are duplex, tmu started, and CP balanced.

3.1. Find out if an OMU is active

CLASSIC

Check that the OMU is ACTIVE

Nortel confidential


(or passive) DUPLEX RNC1204_OMU2:/OMU/local/usr/bin (omu) $ msgViOmuState =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- OMU is MSG_MODULE_ACTIVE OMU is duplex =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

COMPACT On OMU telci telci(omu)> msg omu @ - Result of: msg omu =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- OMU is MSG_MODULE_ACTIVE OMU is duplex =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- telci(omu)> msg omu @ - Result of: msg omu =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- OMU is MSG_MODULE_PASSIVE OMU is duplex =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


3.2. Identify the boards in the CN

CLASSIC RNC1204_OMU2:/OMU/local/usr/bin (omu) $ rbckp listeqp Equipment absPos Shelf Slot Type equipment[0] 0xE5 1 5 OMU equipment[1] 0xE9 1 9 OMU equipment[2] 0xC5 0 5 OMUMMS equipment[3] 0xC6 0 6 MMS equipment[4] 0xC9 0 9 MMS equipment[5] 0xCA 0 10 OMUMMS equipment[6] 0xE7 1 7 CC1 equipment[7] 0xE8 1 8 CC1 equipment[8] 0xEB 1 11 TMU equipment[9] 0xEC 1 12 TMU equipment[10] 0xED 1 13 TMU equipment[11] 0xEE 1 14 TMU equipment[12] 0xCF 0 15 SIM equipment[13] 0xEF 1 15 SIM

COMPACT

There are no more OMU-TMU boards in a compact, as these functions are supported by the AP. Yet, a similar command would be: On OMU telci

Nortel confidential

TELNET_CI> convertAbsPos @ - Result of: convertAbsPos table AbsPos(dec) AbsPos(hex) Instance# LP# AP# =============================================== TMU 193 0xc1 0 2 1 TMU 195 0xc3 1 3 1 TMU 196 0xc4 2 4 1 TMU 203 0xcb 3 5 1 TMU 204 0xcc 4 6 1 TMU 205 0xcd 5 7 1 TMU 206 0xce 6 10 1 TMU 225 0xe1 7 11 1 TMU 227 0xe3 8 12 1 TMU 228 0xe4 9 12 5 TMU 235 0xeb 10 13 1 TMU 236 0xec 11 13 5 TMU 237 0xed 12 14 1 TMU 238 0xee 13 15 1



OMU 229 0xe5 0 4 5 OMU 233 0xe9 1 5 5

3.3. Check that the boards software is up to date

CLASSIC According to the position of the board (shelf and slot) , the bupslice test command allows to check a module software. For example for the TMU whose position has been retrieved previously (rbckp listeqp), shelf 1 slot 11, the board is up to date. RNC1204_OMU2:/OMU/local/usr/bin (omu) $ bupslice test 1 11 +----- Factory ------- +- TM : bs is up to date (cn.tmu.tm.flash.bs.H0 2.5.0.0). +- SBC : devb is up to date (cn.tmu.sbc.flash.firm.H1 1.4.0.0). + bs is up to date (cn.tmu.sbc.flash.bs.H1 2.1.3.0). +----- Normal -------- +- TM : normal is up to date (cn.tmu.tm.flash.norm 2.9.16.0). +- SBC : normal is up to date (cn.tmu.sbc.flash.norm.H1 2.2.14.0). +----- Alternate ----- +- TM : alternate is up to date (cn.tmu.tm.flash.alt 2.9.16.0). +- SBC : alternate is up to date (cn.tmu.sbc.flash.alt.H1 2.2.14.0). ==== The module is up to date ==== The factory parts corresponds to the boot firmware of the board. In the case it wouldnt be up to date, it can be without impact for the TMU behaviour. But normal and alternate modules must be up to date. If not the applications wont be launched on the TMU perceived in a BIST failed state by the Control Node.

COMPACT

On CP (master is by default)

Nortel confidential

8> d sw avl Sw avList = atmNetworking_RI42065, base_RI42065, fabric_RI42065, genericUtilities_RI42065, iRNC_RI42065, ip_RI42065, ipsec_RI42065, networking_RI42065, ss7_RI42065, wanDte_RI42065, wirelessCommon_RI42065, RNCCiph_RI42065055010,



apcBase_RI42065055010, baseExt_RI42065055010, cRNCApc_RI42065055010, cnp_RI42065055010, iRNCApc_RI42065055010, ss7Apc_RI42065055010 ok 2006-04-07 10:21:16.02

3.4. Read the manufacturer and applicative data on a Control Node board

CLASSIC

rbckp getmandata rbckp getappdata For the TM IP address, use the address of the TM board of the active OMU. Its easier to use omu_9_tm or omu_5_tm as these names will be resolved with etc/hosts Example: OMU 9 is active (msgViOmuState), In shelf 1, slot 12 rbckp listeqp highlights a TMU. RNC1204_OMUbis:/OMU/local/usr/bin (omu) $ rbckp omu_9_tm getmandata 1 12 Reading Manufacture data information for module (1,12) PEC code: NTQE56AA Product release: D04 Date of manufacture: FEB19/2002 Serial number: NNTM10178YJ5 RNC1204_OMUbis:/OMU/local/usr/bin (omu) $ rbckp omu_9_tm getappdata 1 12 Reading Application configuration data for module (1,12) Format release: 02 Insertion Date: Oct15/2003 14:43:37 Site: PLATEFORME-BSC3G Unit: 00 Floor: 00 Row Position: ARCY Bay Identifier: CN:UNKNO Shelf: 01 Slot: 12 BIST result:

COMPACT

Nortel confidential

95> d shelf card/1 Shelf Card/1 adminState = unlocked operationalState = enabled usageState = active availabilityStatus = proceduralStatus =



controlStatus = alarmStatus = standbyStatus = providingService unknownStatus = false currentLP = Lp/0 failureCause = none selfTestFault = none sparingConnectionStatus = notApplicable hardwareAlarm = none insertedCardType = CPeE productCode = NTHW08CA-03 pmRevisionCode = NTHW17B2-09 imRevisionCode = N/A serialNumber = NNTM036046FA activeFirmwareVersion = RI42065 inactiveFirmwareVersion = RI41401 memoryCapacity = fastRam : 0 kbyte normalRam : 262144 kbyte sharedRam : 0 kbyte sharedMsgBlockCapacity = 6011 kbyte localMsgBlockCapacity = 800 kbyte timeInterval = 11 minutes cpuUtil = 8 % cpuUtilAvg = 3 % cpuUtilAvgMin = 3 % cpuUtilAvgMax = 3 % memoryUsage = fastRam : 0 kbyte normalRam : 93544 kbyte sharedRam : 0 kbyte memoryUsageAvg = fastRam : 0 kbyte normalRam : 93508 kbyte sharedRam : 0 kbyte memoryUsageAvgMin = fastRam : 0 kbyte normalRam : 93504 kbyte sharedRam : 0 kbyte memoryUsageAvgMax = fastRam : 0 kbyte normalRam : 93533 kbyte sharedRam : 0 kbyte sharedMsgBlockUsage = 0 kbyte sharedMsgBlockUsageAvg = 0 kbyte sharedMsgBlockUsageAvgMin = 0 kbyte sharedMsgBlockUsageAvgMax = 0 kbyte localMsgBlockUsage = 4 kbyte localMsgBlockUsageAvg = 3 kbyte localMsgBlockUsageMin = 3 kbyte localMsgBlockUsageMax = 9 kbyte ok 2006-04-10 13:56:42.00

Nortel confidential



3.5. Check the Control Node led status

CLASSIC

Two leds are visible on the control node boards. Initially three leds were used (rond amber led above the triangle), hidden now, it may still be lid (just after having plugged a board, or during MMS disk accesses). These indications are to be considered with care: the implementation can vary from one load to the other. Green and red leds behaviour OMU/CC1/TMU

Red Led Green Led Status

Module not powered. Bist terminated successfully

The Bist is running or is terminated

unsuccessfully

Passive Module

Active Module

Alarm state Path finding: the module can be removed

Nortel confidential



MMS

Red Led Green Led Status

Nortel confidential


The MMS module is not powered.

The disk is not managed or created.

The MMS is not operational (disk is updating or stopping).

The disk is operational and updated

Alarm state

Path finding: the MMS can be removed

and indicate that the leds are blinking.

COMPACT


Blinking orange LED indicate card is booting.

Red led indicate issue with the board

Green led indicate board is enable active

3.6. Led status from the Control Node consultation

CLASSIC

To read the led status remotely for the CN, use the command: rbckp getled For the TM IP address, use the address of the TM board of the active OMU. Its easier to use omu_9_tm or omu_5_tm as these names will be resolved with etc/hosts. rbckp listeqp gives you the shelf and the slot of a board. RNC1204_OMUbis:/OMU (omu) $ rbckp omu_9_tm getled 1 12 module (1,12) : Led status = 4 => : GREEN : The led status must be converted to a binary on 4 digits and interpreted as follow:

Digit 3 Digit 2 Digit 1 Digit 0

Amber Led (hidden)

Red Led Green Led

If the digit is set to 1 the led is ON. If the digit is set to 0 the led is OFF. In the example given: 4 => 0100 => Green Led is on. If the result was 6 => 0110 => Green and Red leds are on.

Nortel confidential



COMPACT

Nortel confidential


You cannot have the Led status on compact but you can use : d shelf card/* Shelf Card/* Use -noTabular to see the many hidden attributes. +====+-----+----+-----+---------------------+-------+-----+------+------ |Card|osiAd|osiO|osiUs| currentLP |failure|selfT|scstat|hwAlar | | min |per | age | | Cause |estFa| | m | | | | | | | ult | | +====+-----+----+-----+---------------------+-------+-----+------+------ | 0|unlck|ena |activ|Lp/0 |none |none |notApp|none | 1|unlck|ena |activ|Lp/0 |none |none |notApp|none | 2|unlck|ena |activ|Lp/2 |none |none |notApp|none | 3|unlck|ena |activ|Lp/3 |none |none |notApp|none | 4|unlck|ena |activ|Lp/4 |none |none |notApp|none | 5|unlck|ena |activ|Lp/5 |none |none |notApp|none | 6|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 7|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 8|unlck|ena |activ|Lp/8 |none |none |notApp|none | 9|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 10|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 11|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 12|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 13|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 14|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. | 15|unlck|dis |idle | |none |none |notApp|none | |Component is disabled. 9 component(s) reported errors. Component is disable would say that the card is not present in the config. (shelf empty)


3.7. Check if the boards are started, and PEC codes sm_ext_supervision.exe gives SM (Software Management) status for all C-Node modules and most of all their status, PEC code, bist For TMU, OMU and CC1, the status should be operational. For other boards (SIM, MMS) the status will be unknown. If a board is starting, it means it has just reset. Below is an example for few cards (result truncated on purpose). Ctrl-c to exit the tool

Nortel confidential

RNC1204_OMU2:/OMU/local/usr/bin (omu) $ sm_ext_supervision.exe +++ Initialized +++ Sent CN supervision start request. +++ Started - Waiting for supervision information... +++ Received supervision start response at 2005/02/08 14:37:20.644 Response RC: Ok +++ Received supervision report at 2005/02/08 14:37:20.646 Hardware configuration ID = 8 Nu mber of modules present: 14

Position: [Platform: CONTROL NODE; Instance: 0; Sub equip.: MODULE_CA; Shelf: 1; Slot: 5] Hardware Data: Absolute Position : 0xe5 Resource Module type : OMU Hw Release : 1 Hw State : BIST OK Serial Number : NNTM10178NWM PEC Code : NTQE03AB Product Release : 01 Software Data: Status : Operational Managed by OMC : Yes Position: [Platform: CONTROL NODE; Instance: 0; Sub equip.: MODULE_CA; Shelf: 1; Slot: 11] Hardware Data: Absolute Position : 0xeb Resource Module type : TMU Hw Release : 1 Hw State : BIST OK Serial Number : NNTM10178X3T PEC Code : NTQE56AA Product Release : D02 Software Data: Status : Operational Managed by OMC : Yes Position: [Platform: CONTROL NODE; Instance: 0; Sub equip.: MODULE_CA; Shelf: 1; Slot: 15] Hardware Data: Absolute Position : 0xef Resource Module type : SIM Hw Release : 1 Hw State : BIST non significant Serial Number : ?? PEC Code : ?? Product Release : ??



Software Data: Status : Unknown Managed by OMC : Yes If a board is not correctly starting (for example TMU), you may use netstat -r on the active OMU to check that IP packets are sent to it. Editing /etc/hosts file allows determining the kind of board.

COMPACT

Same than for the RNC1000, except that the CC1 doesnt exist anymore. On OMU telci

Nortel confidential


TELNET_CI> sm sm_ext_sup

3.8. Get OMU/TMU FT processes status: ftViCpTable

CLASSIC This command prints the fault tolerant (FT) processes (Core Process or CP) running among OMU and TMU. Here is a result (truncated on purpose), listing the core processes localized on OMU and TMU.

RNC1204_OMU2:/OMU/local/usr/bin (omu) $ ftViCpTable @all CI> CI TTY redirected to: me @all CI> FT_LB_creator[20940] @RNC1204_OMU2 - Result of: cpTable vi *** Printing Core Process Table (22 cp, 14 cg) Omu 0 (cpId = 0,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , FT/LB [Oc: 0 Rk: 1]), (cpId = 1,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , CM_CA [Oc: 0 Rk: 1]), (cpId = 2,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , SupCN [Oc: 0 Rk: 1]), (cpId = 3,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , OAM_CA [Oc: 0 Rk: 1]), (cpId = 4,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , Umts_CP1 [Oc: 0 Rk: 1]), (cpId = 5,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , Umts_CP2 [Oc: 0 Rk: 1]), (cpId = 6,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , Umts_CP3 [Oc: 0 Rk: 1]), (cpId = 7,AbsPos=229, omu, boardNum=0, act.=Act., FtState = Active , Umts_CP4 [Oc: 0 Rk: ), 1]


Tmu 0 (cpId = 16,AbsPos=235, tmu, boardNum = 0, act.=Act., FtState = Active , Umts_Ue [Oc: 0 Rk: 1]), (cpId = 19,AbsPos=235, tmu, boardNum = 0, act.=Act., FtState = Active , Umts_Nodeb [Oc: 0 Rk: 1]), Tmu 1 (cpId = 17,AbsPos=236, tmu, boardNum = 1, act.= Act., FtState = Active , Umts_Ue [Oc: 1 Rk: 1]), (cpId = 20,AbsPos=236, tmu, boardNum = 1, act.= Act., FtState = Active , Umts_Nodeb [Oc: 1 Rk: 1]), Tmu 2 (cpId = 18,AbsPos=237, tmu, boardNum = 2, act.= Act., FtState = Active , Umts_Ue [Oc: 2 Rk: 1]), (cpId = 21,AbsPos=237, tmu, boardNum = 2, act.= Act., FtState = Active , Umts_Nodeb [Oc: 2 Rk: 1]),

2 CP per TMU and 8 CP per OMU must be present. The rank indicates the number of creation of the CP, and therefore can indicate the number of restart of a board. Check that one OMU is holding only active CP and that the other is holding the same CP but in a passive state. Field descriptions: Field Meaning cpId Core process identity

Module absolute position, refer to the following AbsPos

Module type OMU or TMU act FT status FtState Current FT Status of the CP, see below

Unknown the CP has been spawned New The CP is registered to messaging Idle In start-up phase, the CP has loaded its

DP (Permanent Data) from MIB IdlePassive The CP has flushed its DD (Dynamic Data) IdleActive The CP has been swacted active/passive Active CP active Passive CP Passive

CpType Name of the Core Process occ Occurrence of the process rank Rank

Nortel confidential



COMPACT

cpTable used with convertAbsPos. N.B. In V4.2, a new CP (UMTS_CacheMgr) appears on some TMU.

telci(omu)> cpTable Result of: cpTable vi *** Printing Core Process Table (24 cp, 16 cg) Omu 0 (cpId = 0,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , FT/LB [Oc: 0 Rk: 1]), (cpId = 1,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , CM_CA [Oc: 0 Rk: 1]), (cpId = 2,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , SupCN [Oc: 0 Rk: 1]), (cpId = 3,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , OAM_CA [Oc: 0 Rk: 1]), (cpId = 4,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , Umts_CP1 [Oc: 0 Rk: 1]), (cpId = 5,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , Umts_CP2 [Oc: 0 Rk: 1]), (cpId = 6,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , Umts_CP3 [Oc: 0 Rk: 1]), (cpId = 7,AbsPos = 233, omu, boardNum = 0, act.= Act., FtState = Active , Umts_CP4 [Oc: 0 Rk: 1]), Omu 1 (cpId = 8,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , FT/LB [Oc: 0 Rk: 2]), (cpId = 9,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , CM_CA [Oc: 0 Rk: 2]), (cpId = 10,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , SupCN [Oc: 0 Rk: 2]), (cpId = 11,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , OAM_CA [Oc: 0 Rk: 2]), (cpId = 12,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , Umts_CP1 [Oc: 0 Rk: 2]), (cpId = 13,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , Umts_CP2 [Oc: 0 Rk: 2]), (cpId = 14,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , Umts_CP3 [Oc: 0 Rk: 2]), (cpId = 15,AbsPos = 229, omu, boardNum = 1, act.= Pas., FtState = Passive , Umts_CP4 [Oc: 0 Rk: 2]), Tmu 0 (cpId = 16,AbsPos = 193, tmu, boardNum = 0, act.= Act., FtState = Active , Umts_Nodeb [Oc: 0 Rk: 1]), (cpId = 19,AbsPos = 193, tmu, boardNum = 0, act.= Act., FtState = Active , Umts_Ue [Oc: 0 Rk: 1]), Tmu 1 (cpId = 17,AbsPos = 196, tmu, boardNum = 1, act.= Act., FtState = Active , Umts_Nodeb [Oc: 1 Rk: 1]), (cpId = 20,AbsPos = 196, tmu, boardNum = 1, act.= Act., FtState = Active , Umts_Ue [Oc: 1 Rk: 1]), Tmu 2 (cpId = 18,AbsPos = 203, tmu, boardNum = 2, act.= Act., FtState = Active , Umts_Nodeb [Oc: 2 Rk: 1]), (cpId = 21,AbsPos = 203, tmu, boardNum = 2, act.= Act., FtState = Active , Umts_Ue [Oc: 2 Rk: 1]), Tmu 3 (cpId = 22,AbsPos = 195, tmu, boardNum = 3, act.= Act., FtState = Active , Umts_Ue [Oc: 3 Rk: 1]), (cpId = 23,AbsPos = 195, tmu, boardNum = 3, act.= Act., FtState = Active , Umts_CacheMgr [Oc: 0 Rk: 1]), WARNING: the boardNumber attribute should not be confused with the instance number. The table below provides that conversion AbsPos(dec) AbsPos(hex) Instance# LP# AP# =============================================== TMU 193 0xc1 0 2 1 TMU 195 0xc3 1 3 1 TMU 196 0xc4 2 4 1 TMU 203 0xcb 3 5 1 TMU 204 0xcc 4 TMU 205 0xcd 5 TMU 206 0xce 6 TMU 225 0xe1 7 TMU 227 0xe3 8 TMU 228 0xe4 9 TMU 235 0xeb 10 TMU 236 0xec 11 TMU 237 0xed 12 TMU 238 0xee 13 OMU 229 0xe5 0 4 5 OMU 233 0xe9 1 5 5

Nortel confidential



3.9. Get OMU/TMU/CC1 non FT processes status

CLASSIC sm_launcher.exe board ps This command gives the status of non FT processes groups and processes. See chapter 3.11. OMU and TMU board address according to the Module Absolute Position to identify a board absolute position. All processes groups should be executing, all the processes should be started or finished. Example for OMU 9 / abs pos 489 / fully started (truncated result on purpose) RNC1204_OMU2:/OMU/local/usr/bin (omu) $ sm_launcher.exe board 489 ps Board position set to: [Platform: CONTROL NODE; Instance: 0; Sub equip.: SBC; Shelf: 1; Slot: 9] RC: Ok Report: Size: 3921 [octets]. Text follows. Process Group State Time Launched Activity_Election_Platform Executing 2005/02/09 16:43:27.331 Minimal_FT_Platform_1 Executing 2005/02/09 16:44:00.495 Minimal_Non_FT_Platform Executing 2005/02/09 16:43:43.427 Oam_Minimal_Platform Executing 2005/02/09 16:43:26.038 UMTS_FT_Platform Executing 2005/02/09 16:45:31.330 UMTS_Non_FT_Platform Executing 2005/02/09 16:45:35.626

Nortel confidential



PID Application Name State Instance 5162 FM Started 0 2916 SWB Started 0 15998 ACTIVITY_MNGT Started 0 11926 OAM_HM Started 0 15762 APE Started 0 14218 ATM_OMU_Selection Started 0 18602 CM_LA Started 0 18864 DAS Started 0 12690 FAS Started 0 16564 FT_LA Started 0 12962 FmBosEvent Started 0 18362 Histo Started 0 18098 OamOvCa Started 0 19350 SCI_CA Started 0 19608 SCI_LA Started 0 19866 TEST_DIAG_CN_MGNT Started 0 20382 TEST_DIAG_SERVER Started 0 20640 UpgradeCn Started 0 20898 UpgradeLmt Started 0 19110 CREATOR_CMCA Started 0 17896 FT_LB_creator Started 0 21416 OAM_CA Started 0 20148 OamCn_Creator Started 0 21672 OamSmCn_crtr Started 0 21930 SwbCreator Started 0 22966 ATMM_CA_CRE Started 0 13970 CPM_CREATOR Started 0 5764 CREATOR_CMCA_H Started 0 23248 DDR_CREATOR Started 0 22714 IN_CREATOR Started 0 17326 IUXADM_CREATOR Started 0 13700 IU_CREATOR Started 0 22220 PFCA_CREATOR Started 0 23478 TR_CA_CREATOR Started 0 23736 FMGSC_CA Started 0 23994 FMGSC_MA Started 0 24252 IN_Access_LA Started 0 24510 IN_PROXIES Started 0 24768 PFPROXY Started 0 25026 RNCUPGSUP Started 0 25284 TR_PROXY Started 0

Nortel confidential



COMPACT The menu is the same as for an RNC1000. On OMU telci

Nortel confidential


TELNET_CI>sm sm_launcher @ - Result of: sm sm_launcher SM Launcher =========== Operating on local board: [Platform: CONTROL NODE; Instance: 0; Sub equip.: SBC; Shelf: 1; Slot: 9] Command Description ------- ----------- create Create a process group modify Modify a process group delete Delete a process group get_create_data Get process group create data ps Process status pgs Process group status bs Board status board Set board position board-shutdown Board shutdown oam-shutdown OAM shutdown quit Quit Command: The menu offers the same facilities than for the classic. ps dumps the process status on the OMU, to check that all the processes are launched and running.


3.10. Find out if spares TMU are present

CLASSIC

rbckp listeqp will list all the plugged TMU in the Control Node sm_ext_supervision.exe allows the starting or the operational state of the

boards plugged. the spare TMU wont be listed in ftViCpTable: they wont hold any Core

Process. For a fully fitted Control Node (14 tmu plugged) 2 TMU are spare. For a 4 TMU configuration one TMU should be spare.

COMPACT

Not applicable.

3.11. OMU-TMU board address according to the Module Absolute Position

CLASSIC

Be sure to differentiate module absolute position and board absolute position You can either

translate its absolute position in hex, refer to rbckp listeqp result (see chapter 2. How to identify the boards in the CN) and use /etc/hosts to identify the board name resolution

calculate directly its ip address, First way

For example, for TMU 0 in ftViCpTable result, its absolute position is 235 1. 235 converted in hexa is 0xEB 2. rbckp listeqp result was : Equipment absPos Shelf Slot Type equipment[8] 0xEB 1 11 TMU 3. The tmu can be reached by opening a telnet command interpreter session on the board:

Nortel confidential



Nortel confidential


telci tmu_1_11_sbc for the sbc board telci tmu_1_11_tm for the tm board The name resolution is assumed by /etc/hosts

Or 2nd way

1. Convert the AbsPos in hex 2. Mask it with 0x2F (AbsPos AND 0x2F) 3. Convert it in decimal 4. The IP addresses are: TMU-TM: 172.decimalvalue.2.1 TMU-SBC: 172.decimalvalue.2.2 Example Absolute position 235 235 0xEB 0xEB AND 0x2F = 0x2B 0x2B 43

TMU-TM: 172.43.2.1 TMU-SBC: 172.43.2.2

The C-Node boards internal IP addresses, names and absolute positions can also be find in the file /etc/hosts. TMU internal boards names, internal boards addresses, absolute boards positions and absolute modules positions:


Nortel confidential


Shelf 1 Slot 1 Slot 3 Slot 4 Slot 11 Slot 12 Slot 13 Slot 14

Module absolute position

0xe1 (225)

0xe3 (227)

0xe4 (228)

0xeb (235)

0xec (236)

0xed (237)

0xee (238) TMU TM (name of the board, IP address, board absolute position)

tmu_1_1_tm

172.33.2.1

0x2e1 (737)

tmu_1_3_tm

172.35.2.1

0x2e3 (739)

tmu_1_4_tm

172.36.2.1

0x2e4 (740)

tmu_1_11_tm

172.43.2.1

0x2eb (747)

tmu_1_12_tm

172.44.2.1

0x2ec (748)

tmu_1_13_tm

172.45.2.1

0x2ed (749)

tmu_1_14_tm

172.46.2.1

0x2ee (750)

TMU SBC (name of the board, IP address, board absolute position)

tmu_1_1_sbc

172.33.2.2

0x1e1 (481)

tmu_1_3_sbc

172.35.2.2

0x1e3 (483)

tmu_1_4_sbc

172.36.2.2

0x1e4 (484)

tmu_1_11_sbc

172.43.2.2

0x1eb (491)

tmu_1_12_sbc

172.44.2.2

0x1ec (492)

tmu_1_13_sbc

172.45.2.2

0x1ed (493)

tmu_1_14_sbc

172.46.2.2

0x1ee (494)

Shelf 0 Slot 1 Slot 3 Slot 4 Slot 11 Slot 12 Slot 13 Slot 14


0xc1 (193)

0xc3 (195) 0xc4 (196) 0xcb (203) 0xcc (204) 0xcd (205) 0xce (206)

TMU TM (name of the board, IP address, board absolute position)

tmu_0_1_tm

172.1.2.1

0x2c1 (705)

tmu_0_3_tm

172.3.2.1

0x2c3 (707)

tmu_0_4_tm

172.4.2.1

0x2c4 (708)

tmu_0_11_tm

172.11.2.1

0x2cb (715)

tmu_0_12_tm

172.12.2.1

0x2cc (716)

tmu_0_13_tm

172.13.2.1

0x2cd (717)

tmu_0_14_tm

172.14.2.1

0x2ce (718)

TMU SBC (name of the board, IP address, board absolute position)

tmu_0_1_sbc

172.1.2.2

0x1c1 (449)

tmu_0_3_sbc

172.3.2.2

0x1c3 (451)

tmu_0_4_sbc

172.4.2.2

0x1c4 (452)

tmu_0_11_sbc

172.11.2.2

0x1cb (459)

tmu_0_12_sbc

172.12.2.2

0x1cc (460)

tmu_0_13_sbc

172.13.2.2

0x1cd (461)

tmu_0_14_sbc

172.14.2.2

0x1ce (462)


Slot 5 Slot 6 Slot 7 Slot 8 Slot 9 Slot 10


0xe5 (229) 0xe7 (231) 0xe8 (232) 0xe9 (233)

TM (name of the board, IP address, board absolute position) omu_5_tm omu_9_tm

172.37.2.1 172.41.2.1

0x3e5 (997) 0x3e9 (1001)

PMC (name of the board, IP address, board absolute position)

omu_9_pmc1 omu_5_pmc1

172.41.3.2 172.33.3.2

0x3e9 (1001) 0x3e5 (997)

SBC (name of the board, IP address, board absolute position)

omu_9_sbc cc1_8_sbc cc1_7_sbc omu_5_sbc

172.41.2.2 172.253.1.40172.253.1.39172.37.2.2

0x1e9 (489) 0x1e8 (488) 0x1e7 (487) 0x1e5 (485)

To check messaging traffic between all SBC boards msgFullMeshTable On an OMU-SBC board, traffic should be opened towards :

its OMU-TM board both CC1-SBC boards all the TMU-SBC boards.

RNC1204_OMU2:/OMU/local/usr/bin (omu) $ msgViFullMeshTbl [0] [1] [2] [3] [4] [5] [6] [7] omu_3 cnbos This command checks:

Nortel confidential



the IP/ATM connection between OMU-SBC and each card (TM,SBC) that the minimal software is started on SBC boards.

The result must be ok. RNC1204_OMU2:/OMU/local/usr/bin (omu) $ omu_diag 3 cnbos ++ cnnet ++ Control-Node IP Access ++++++++++++++++++++++++++ - ping 172.253.1.37--- 1 5- OMU- TM................. ok - ping 172.37.2.2----- 1 5- OMU- SBC................ ok - ping 172.253.1.41--- 1 9- OMU- TM................. ok - ping 172.41.2.2----- 1 9- OMU- SBC................ ok - ping 172.253.1.39--- 1 7- CC1- SBC................ ok - ping 172.253.1.40--- 1 8- CC1- SBC................ ok - ping 172.253.1.43--- 1 11 TMU- TM................. ok - ping 172.43.2.2----- 1 11 TMU- SBC................ ok - ping 172.253.1.44--- 1 12 TMU- TM................. ok - ping 172.44.2.2----- 1 12 TMU- SBC................ ok - ping 172.253.1.45--- 1 13 TMU- TM................. ok - ping 172.45.2.2----- 1 13 TMU- SBC................ ok - ping 172.253.1.46--- 1 14 TMU- TM.................

rnc and nodeb health check

Documents