5 troubleshooting & maint

105
1 Troubleshooting and Maintenance HUGHES PROPRIETARY II Hughes Network Systems AiReach Broadband 9000 Troubleshooting and Maintenance Training Course September 2011

Upload: derrick-senyo

Post on 15-Jan-2016

64 views

Category:

Documents


0 download

DESCRIPTION

Hughes radio operations and maintenance

TRANSCRIPT

Page 1: 5 Troubleshooting & Maint

1 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Hughes Network Systems

AiReach Broadband 9000

Troubleshooting and Maintenance

Training Course

September 2011

Page 2: 5 Troubleshooting & Maint

3 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Reference Documents

Hub and Remote Terminal Site Preparation Guide Hub Terminal Installation and User Manual Remote Terminal Installation and User Manual EMS Reference Manual System Operations Manual Element Management System (EMS) Installation and

Maintenance Manual

All Manuals are online on the EMS and on the EMS software CD-ROM

Page 3: 5 Troubleshooting & Maint

4 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Course Topics

Troubleshooting– Isolating Problems– Installation Troubleshooting– System Maintenance– Operational Troubleshooting

Support Contact– TAC Problem Reporting– TAC Website Use

Page 4: 5 Troubleshooting & Maint

5 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Before You Start

PC Settings for LCT (Local Craft Terminal)– 9600, 8, none, 1– No handshaking

LCT Cable:

LCT Login– Username: lct– Password: hnslct

7

6

5

4

3

2

6

5

4

3

2

1

9

8

7

Correct LCT Cable

4 wire, 8 positionmodular

plug

DB-9FBackshell

Correct N/C

N/CN/C = No Connection

N/C

N/C

N/C

N/C

N/C

N/C

N/C

1

8

6

5

4

3

2

1

4 wire, 6 positionmodular

plug

N/C

N/C

N/C

N/C

Page 5: 5 Troubleshooting & Maint

6 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Troubleshooting

Page 6: 5 Troubleshooting & Maint

7 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Troubleshooting Tree

Troubleshooting Categories

InstallationAnd

Commissioning

OperationsAnd

Maintenance

For each category above:– The starting points are different, but– Some of the troubleshooting techniques overlap

Page 7: 5 Troubleshooting & Maint

8 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Isolating Problems Most problems identified during installation are more

likely to be physical/mechanical defects, problems created by human error at the time of the component configuration at the EMS, or parameters entered in the IDU during commissioning.

Most problems identified during normal operation tend to be categorized as network, RF, or specific service related failures that are recognized by the current alarms reported to the EMS.

It is possible to experience operational problems that may normally be classified as occurring during installation, and vice versa.

Regular system maintenance can help to eliminate some problems by identifying them as they develop rather than after they become a service impact.

Page 8: 5 Troubleshooting & Maint

9 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Installation Troubleshooting

Page 9: 5 Troubleshooting & Maint

10 Troubleshooting and Maintenance HUGHES PROPRIETARY II

HT Installation Troubleshooting

a) Power On

b) Start Up Diagnostics (SUDs)

c) Bootp request failed

d) Disabled Failed

e) Disabled Off Duty

f) Enabled Degraded

Page 10: 5 Troubleshooting & Maint

11 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Power On

Upon powering up the IDU the power LED should be solid green color

Verify input power is within the operating range:– +/- 48vdc: +/- 40.5vdc to +/- 57vdc

Does rectifier have enough current capability? If the IDU does not power on, remove the IFL cable

and try again. If it powers on, there is a short in the IFL cable.

Page 11: 5 Troubleshooting & Maint

12 Troubleshooting and Maintenance HUGHES PROPRIETARY II

SUDs (Start Up Diagnostics)

A failure during SUDs indicates one of the IDU self tests has failed.

If there is a failure the “status LED” will remain off, or be illuminated a solid color, and the LCT can not be accessed.

If the LCT is not accessible and the status LED is flashing, check your HyperTerminal settings (9600, 8, N, 1) and your serial cable.

Replace the IDU if SUDS fails

Page 12: 5 Troubleshooting & Maint

13 Troubleshooting and Maintenance HUGHES PROPRIETARY II

HT Bootup

On the PC LCT you should see the following (or similar) after power up.

It can take about 20 seconds to see some output from the IDU.

ROM_DISK_START address = 0xfe600000 ,size = 20971520ROMDISK OK. checking BSDBSD present in the ROMDISK

VxWorks System Boot

Copyright 1984-1998 Wind River Systems, Inc.

CPU: Motorola - PowerPC 8280Version: 5.4.2BSP version: 1.2/2Creation date: Suppressed Date/Time

Page 13: 5 Troubleshooting & Maint

14 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Bootp Request Fails

After exiting the “install mode” the HT will send a boot request to the EMS

booting...

boot device : lnunit number : 0 processor number : 0 file name : /aireach/ems/DBA/SWREL/R011307/HJBA1307.CPZinet on ethernet (e) : 10.10.10.10:ffffff00host inet (h) : 10.10.10.1gateway inet (g) : 10.10.10.1user (u) : anonftp password (pw) : anonflags (f) : 0x40

..........Bootp request failed after 10 retries...

Page 14: 5 Troubleshooting & Maint

15 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Bootp Request Fails

If after a few minutes the message “Bootp request failed after 10 retries” is received, there was no response from the EMS

Perform the following:– Verify HT MAC on EMS Hub Terminal screen.– On the hub site router:

• verify the “Helper Address” is defined to point the bootp broadcast message to the EMS IP address.

– Verify Network Management Connectivity (next slide)

Page 15: 5 Troubleshooting & Maint

16 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Verify Network Management Path

Verify LAN cable is connected to LAN port 3 on the 9800 IDU Verify the path to the hub router from the EMS by using the

“ping” command. Verify the path using “trace route” command. Determine if a point between the EMS and HT is not reachable. If the problem is in the DCN contact the DCN support staff. If the last router to the hub site is reachable, a site visit may be

required to verify the device interface cable integrity, or to verify the device is correctly configured.

TIP: You can use your laptop to verify connectivity from the Hub site (or router location). Set the IP of your laptop to the IP of any HT on the site. See if you can ping the near router interface, far router interface and the EMS IP address. Also, ask the EMS Operator to ping your laptop.

Page 16: 5 Troubleshooting & Maint

17 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Verify Network Management Path

If you are using E1 TDM in-band management:– Use the LCT to show the status of the E1 Physical link

– Check the E1 physical link status on the router

– Verify that the number of timeslots matches

– If there are any E1 alarms, the HT will not be able to boot

– While the HT is in the process of rebooting (after trying to get the boot reply from the EMS), the router will show LOS on the E1 line… this is normal. The E1 is only active when it is trying to send and receive the bootp messages.

If you are using ATM in-band management:– Use the LCT to show the status of the STM-1/E3 Physical link

– Check the STM-1 physical link status on the router or switch

– Verify that the management VP/VC is correct

Page 17: 5 Troubleshooting & Maint

18 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Disabled Off Duty

========================================================================== AB9800 Hub Terminal Main Menu Main MenuLCT Mode:Normal[RAM], PtP Mode:OFF, PtP Install Mode:OFFLocal Config Install:OFF, Local Config Normal:OFF, EMS Connectivity:LAN========================================================================== 1. Front Panel Display 2. Site Specific Information 3. EMS Port Information 4. Status 5. Statistics 6. Installation 7. Reset History 8. Reset 9. Software Revision 10. Set/Clear Configuration Mode 11. Local Configuration 12. Baud Rate Setting 13. Update Base Application in Flash 14. Update Boot ROM in Flash 15. Update FPGA in Flash 16. Shutdown 17. Logout

Enter your choice from the menu, OR 'p' for previous menu, 'm' for main menuYour Choice: 1

========================================================================== Front Panel Display========================================================================== HT Freq Disabled OffDutyPress Enter to continue...

Page 18: 5 Troubleshooting & Maint

19 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Disabled Off Duty

Disabled Off Duty indicates the IDU has not received all configuration from the EMS yet: it’s either in process of downloading it or it failed to complete FTP process with the EMS.

To find out if this is the case, logon to LCT and check “Status” – “Element Software”. If state is listed as “Unknown” or message says “Reconciling” and it stays that way for more than 10 minutes – most likely that FTP process has failed.

Check router configuration – make sure that:– “ip directed broadcast” and – “ip irdp” are setup on the Hub Router interface.

Check if company firewall or router access list maybe enabled and blocking FTP/ IPs range

Page 19: 5 Troubleshooting & Maint

20 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Disabled Failed

With the HT successfully downloaded from the EMS, (“Status” – “Element Software” shows “Reconciled” state) the final state shows:– HT Freq XX– Disabled Failed

Use the “Status option” on the LCT menu to identify the hardware failure. If ODU is failed…replace ODU. If IDU is failed…replace the IDU.

Page 20: 5 Troubleshooting & Maint

21 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Enabled Degraded

Enabled Degraded indicates the IDU is Enabled and transmitting, but one of the configured service ports does not have an interface cable installed or is not configured correctly.

Page 21: 5 Troubleshooting & Maint

22 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Other Issues

Sometimes if all other indications are correct, it is possible that the EMS database needs to be refreshed. On the EMS, the operator must run this command:– $ DescriptorRegenAll.ksh

This will force the HT configuration files to be rewritten and downloaded

Note: Please, be aware that if, for some reason, there were no communication between the EMS and HUB(s) at the time when configuration changes have been made at the EMS – changes will take immediate effect upon execution above command and/or when communication has been restored

Page 22: 5 Troubleshooting & Maint

23 Troubleshooting and Maintenance HUGHES PROPRIETARY II

RT Installation Troubleshooting

a) Power On

b) Start Up Diagnostics (SUDs)

c) Unable to read Frequency set

d) Stuck at Freq Acq in progress

e) Stuck at Ranging in progress

f) Install Acq Denied HT #

g) Bootp Request Failed

h) Disabled Failed

i) Disabled Off Duty

j) Enabled Degraded

k) Matching RF Planning data

Page 23: 5 Troubleshooting & Maint

24 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Power on

Upon powering up the IDU the power LED should be a solid green color

Verify input power is within the operating range:– +/- 48vdc: +/- 40.5vdc to +/- 57vdc

For DC power does rectifier have enough current capability?

If the IDU does not power on, remove the IFL cable and try again. If it powers on, there is a short in the IFL cable.

Page 24: 5 Troubleshooting & Maint

25 Troubleshooting and Maintenance HUGHES PROPRIETARY II

SUDs (Start Up Diagnostics)

A failure during SUDs indicates one of the IDU self tests has failed.

If there is a failure, the “status LED” will remain off or a solid color and the LCT can not be accessed.

If this happens, replace the IDU (note when replacing the IDU make sure to contact the EMS operator and report the new MAC address).

Page 25: 5 Troubleshooting & Maint

26 Troubleshooting and Maintenance HUGHES PROPRIETARY II

IDU Hangs After Entering Cable Loss

This occurs while the IDU is in the “install mode”. The LCT is stuck after inputting IFL cable/loss information and does not show the RT Frequency selection.

Enter your choice from the menu Your Choice(1-14): 1

========================================================================== IFL Cable Mode========================================================================== 1. Cable Loss 2. Cable Type/Len

Enter your choice from the menu Current Choice: 1Your Choice(1-2): 1

Enter the 'Rx Loss (0.1 dB)' default = 47 :

Enter the 'Tx Loss (0.1 dB)' default = 67 :

IDU Hangs

Page 26: 5 Troubleshooting & Maint

27 Troubleshooting and Maintenance HUGHES PROPRIETARY II

IDU Hangs After Entering Cable Loss

This indicates the IDU was unable to poll the ODU to verify it is the correct band radio for the frequency selected. This may be due to:– an open connection between the IDU and ODU…verify the

integrity of the IFL cable, and verify the IFL connections at the IDU and ODU have been properly made;

– HT ODU was mistakenly installed at RT site… verify for the correct ODU part number

– Unsupported ODU – Check the part numbers– Faulty ODU (rare) – replace ODU

If after inputting IFL cable loss/type information output screen displays wrong frequency set ( expected, for example A1-A20, displayed B1-B20) this indicates that the wrong band RT ODU has been installed at the site. Verify for the correct RT ODU by part number.

Page 27: 5 Troubleshooting & Maint

28 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Stuck at “Frequency Acq in progress”

========================================================================== Select 26C Freq========================================================================== 1. C1 2. C2 3. C3 4. C4 5. C5 6. C6 7. C7 8. C8 9. C9

Enter your choice from the menu Your Choice(1-9): 1

---------------------------------------- Freq Set C1 ----------------------------------------

---------------------------------------- Freq Acq In Prog---------------------------------------- Stays here for a

long time > 15 minutes

Page 28: 5 Troubleshooting & Maint

29 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Stuck at “Frequency Acq in progress” This indicates the RT is unable to synchronize to the HT.

Verify there are no other RTs having a problem in this sector. If this is the first RT in the sector, verify the HT is operational.

Another possible cause may be that the RT boresight alignment was performed incorrectly and the RT is not pointed well enough receive on either side lobe, or the main lobe. Verify the RT pointing.

If it is a point to point link, verify the HT pointing. It is possible the IFL cable integrity is not proper and a

poor quality IF signal is getting to the IDU. Verify the IFL cable sweeps and/or IFL cable connectors ( loose, improperly terminated)

(continued...)

Page 29: 5 Troubleshooting & Maint

30 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Stuck at “Frequency Acq in progress”

... Verify that the proper RT antenna is used (30cm used

instead of 60cm). Verify that the RT ODU is not installed with the incorrect

polarization Verify that correct Polarity antenna is installed at the Hub

Terminal Another possible cause is the ODU or IDU is failed…

replace the ODU or IDU

Page 30: 5 Troubleshooting & Maint

31 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Stuck at “Ranging in progress”

========================================================================== Select 26C Freq========================================================================== 1. C1 2. C2 3. C3 4. C4 5. C5 6. C6 7. C7 8. C8 9. C9

Enter your choice from the menu Your Choice(1-9): 1

---------------------------------------- Freq Set C1 ----------------------------------------

---------------------------------------- Freq Acq In Prog----------------------------------------

---------------------------------------- Ranging In Prog----------------------------------------

Stays here for a long time > 15

minutes

Page 31: 5 Troubleshooting & Maint

32 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Stuck at “Ranging in progress”

Ranging is performed at the demodulator in the IDU. The RT is able to receive signal from the HT well enough to synchronize, but there are too many errors on the link to perform the ranging process. Verify no other RTs in the sector are having problems. If so, check the HT site.

Since at this point the pointing voltage port is enabled on the ODU verify the RT antenna pointing to improve the receive signal quality.

Verify in the RF Planning data if a mechanical attenuator is required at this site. If so, verify it has been installed.

Verify the IFL cable sweeps.

If the link is Point to Point– Verify that you exited install mode at the HT– Verify that the range is set correctly

Page 32: 5 Troubleshooting & Maint

33 Troubleshooting and Maintenance HUGHES PROPRIETARY II

“Install Acq Denied HT#”

Install acquisition denied indicates that the RT ESN was not known by the HT. Verify the HT ID corresponds to the expected serving HT, and if so, check with the EMS operator that the ESN was properly input into the EMS database.

Another possible cause may be that, if this is a RT site on the sector fringe, the HT ID denying acquisition is not the expected serving HT but may be another HT operating on the same frequency as the desired HT. If so, the HT antenna may not be pointed properly to give the correct coverage footprint.

Page 33: 5 Troubleshooting & Maint

34 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Bootp Request Failed If the RT has successfully been taken out of the install

mode yet there is a failure to receive a reply to the bootp request, then the air link quality or the IF quality is suspect.

Put the RT back into the installation mode and repoint the antenna.

Review the actual IFL cable sweeps paying particular attention to the sweep cable characteristics.

Check the SQI – there may be interference on the link Check the Interface option on the EMS If the antenna pointing and IFL appears correct replace

the IDU (note when replacing the IDU make sure to contact the EMS operator and report the new ESN).

Page 34: 5 Troubleshooting & Maint

35 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Disabled Off Duty

Disabled Off Duty indicates the IDU has not received all software from the EMS yet: it’s either in process of downloading it or it’s failed to complete FTP process with the EMS.

To find out if this is the case, logon to LCT and check “Status” – “Element Software”. If state is listed as “Unknown” or message says “Reconciling” and it stays that way for more than 10 minutes – most likely that FTP process has failed.

Check EMS configuration ( subnet mask), check router configuration.

Check HT/RT Air Interface status – FTP process can get interrupted due to poor air link quality.

Page 35: 5 Troubleshooting & Maint

36 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Disabled Failed

With the RT successfully downloaded from the EMS, the final state shows:– RT Freq XX– Disabled Failed

Use the “status option” on the LCT menu to identify the hardware failure. If ODU is failed…replace ODU. If IDU is failed…replace the IDU.

Page 36: 5 Troubleshooting & Maint

37 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Enabled Degraded

Enabled Degraded indicates the IDU is Enabled and transmitting, but one of the configured service ports does not have an interface cable installed or is not configured correctly.

Page 37: 5 Troubleshooting & Maint

38 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Matching RF Planning Data

It is important to compare the RF operational characteristics of the site to the site RF plan.

Compare actual values of RSSI and Tx power to the RF Plan The actual RSSI keeping in mind that the RSSI displayed initially after the

install may be based on only the QPSK burst (messaging burst between HT and RTs). Unless this site or another RT is configured for the same modulation and a service is also configured on the sector, the RSSI is based on the QPSK burst only (the LCT indicates if the measurement is taken from a QPSK burst).

When comparing RSSI and Tx power, it is important to remember the accuracy of the values shown on the LCT, or reported on the EMS is affected by inherent variables in the system.

These variables include IFL cable loss, HT Tx power, specific ODU RF characteristics, and environmental conditions at the time (RF Planning data is usually shown based on clear sky conditions). It is fair to assume if actual RSSI and RF Planning data are comparable within +/-5dB then things are ok. If Tx power is within a few dB then it is ok to continue.

Page 38: 5 Troubleshooting & Maint

39 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Link Balancing

It’s a good idea to see if a link is “balanced” Balance means that uplink path loss closely match

downlink path loss A mismatch indicates a problem most likely in cable loss

values, IFL cable or could indicate an ODU problem

Uplink path loss = abs ( RT TX pwr – RSSI @ HT )

Downlink path loss = abs ( HT Tx pwr – RSSI @ RT )

HT RT

RSSI from RT

HT TX Power

RT TX Power

RSSI from HT

Uplink

Downlink

Page 39: 5 Troubleshooting & Maint

40 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Enhanced LCT – Normal Mode

Provides the ability to debug RF issues from the HT LCT (without EMS access)

“LCT Local Configuration” must be enabled on the EMS

If enabled, the HT LCT can be used to adjust:– Frequency (Channel)

– Clock Source

– ODU QPSK Transmit Power

– ODU 16-QAM Transmit Power

– Target RSSI

When disabled at the EMS, all values return to the EMS Configuration values

Page 40: 5 Troubleshooting & Maint

41 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Operational Troubleshooting

Page 41: 5 Troubleshooting & Maint

42 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Troubleshooting – Bottom Up

CPENetwork

Network Managment

HT Hardware

Port(s)

RT Hardware

Port(s)

Air Interface

Services

ODU ODU

Always start looking at the problem from the bottom up.

For example, if a service has errors you may:

1. Check hardware first – IDU, ODU

2. Check physical layer – ports, air interface

3. Check services

Page 42: 5 Troubleshooting & Maint

43 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Troubleshooting General Steps

Look at the following in the order shown:

– Current Alarms• Can the site be reached from the EMS? • If not, is it one HT or all?

– Alarm Log (Log History)

– HT/RT Summary Status

– HT/RT Detail Status – Look at overall status to see a failure has occurred

– HT/RT Air Interface – Air Interface must be good before looking at anything else

– HT/RT Port Status – Look at interface state, alarm bits, duplex mode, etc as applicable

– HT/RT Port & Service statistics – Look at cell/frame counts, ingress & egress errors, etc.

– Use Loopbacks to isolate problems

Page 43: 5 Troubleshooting & Maint

44 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Current Alarms

Page 44: 5 Troubleshooting & Maint

45 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Current Alarms

The current alarm screen identifies real time faults in the system.

Alarm severities range from:– Warning - Blue colored alarm showing details of

potential problems– Minor – Yellow colored alarm showing details of

system alarms that do not affect services– Major – Orange colored alarm showing details of

system alarms that could affect, or do affect services– Critical – Red colored alarm indicating details of

alarms having a major impact on services.

Page 45: 5 Troubleshooting & Maint

46 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Alarm Types

Communication alarms – Result from errors on the internal or external communication interfaces of the AB9000 network.

Quality of Service alarms – Related to service connections.

Equipment alarms – Result from equipment hardware or software failures.

Page 46: 5 Troubleshooting & Maint

47 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Acknowledged and Cleared Alarms

Each operator should track open alarms daily to monitor the progress of technical personnel dispatched to clear site faults.

Once an alarm is being worked on it should be acknowledged.

Admin Note: With multiple operators having the same privileges, the acknowledged alarm log should be reviewed by the system administrator to verify that all cleared alarms had the appropriate attention provided and fault resolutions implemented.

Page 47: 5 Troubleshooting & Maint

48 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Topology View - Alarms

The Alarm Summary shows ALL alarms from

the selected node, and below

The nodes are colored according to the highest severity

alarm on that node AND its children

Quick Navigation to debugging

screens: Detail Status, Alarms,

Summary Status

Link States:

Green line: Normal connected state (Summary Status is ok)

Orange line: Partial NM connectivity

Red dash line: not connected

Page 48: 5 Troubleshooting & Maint

49 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Current Alarm Screen

The Fault Element is the element of subcomponent where the fault actually is

The Reporting Element is the one where the alarm actually

came from

Sometimes they are the same. In this example,

the EMS is reporting that if can’t contact an RT (check the air link!)

Page 49: 5 Troubleshooting & Maint

50 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Current Alarms Details Screen

Current alarm details lead the operator to the fault element. These details combined with the use of status, statistics, and loopbacks help for quick drill down and fast problem

resolution.

Direct access to:Configuration

Details Status & ControlSummary Status

Page 50: 5 Troubleshooting & Maint

51 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Alarms Log

All Alarms are recorded in the Alarm Log to keep a historical record of Alarms

All Acknowledge and Clears are recorded

Alarm Log can be filtered to provide, for example, history of a particular service or element

Page 51: 5 Troubleshooting & Maint

52 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Operational TroubleshootingHT and RT Summary Status

Page 52: 5 Troubleshooting & Maint

53 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Summary Status – Hub Terminal1. Select the Hub

2. Press Read

Operational and Availability States

show you the current status for the HT

Admin State is the state configured

on the HT Configuration

screen

This shows you if a redundant HT is

switched in or switch out

Here you can see the last time the EMS

received a “heartbeat” message

from the HT

Page 53: 5 Troubleshooting & Maint

54 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Summary Status - Remote Terminal

By highlighting a row and clicking on the button ( to the right ), operator can “jump” and review more detailed data

1. Select the Hub and Sector

2. Press Read

Operational and Availability States

show you the current status for the RT

This shows you if a redundant RT is

switched in or switch out

Here you can see the last time the EMS

received a “heartbeat” message

from the RT

If Adaptive Modulation is in use,

you can see the configured and

current modulations

Page 54: 5 Troubleshooting & Maint

55 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Operational TroubleshootingHT and RT Detail Status

Page 55: 5 Troubleshooting & Maint

56 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Ping, Trace Route

Access these dialog boxes from HT and RT Detail

The screens are very useful

to determine if an HT or RT

is reachable...

... If not, how far into the network can we

reach? Is it a router or network

issue?

Page 56: 5 Troubleshooting & Maint

57 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Hub Terminal Detail Status & Control

This is a switched in Backup HT.

Hub Terminal can be reset from the EMS

Hub Terminal can be switched to the

primary or backup HT

The main tab provides an overview of current status

of HT components

Page 57: 5 Troubleshooting & Maint

58 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Hub Terminal Detail Status – Air InterfaceRT Status shows

Modulation...

... and Uplink and Downlink RSSI and Tx Power

Entered Cable Loss values

Channel and bandwidth

ODU Type

Page 58: 5 Troubleshooting & Maint

59 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Hub Terminal Detail Status & Control

Current Alarm tab displays all alarms existing at the HT at

this moment. Probable cause and additional text would help

to resolve the issue.

Page 59: 5 Troubleshooting & Maint

60 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Remote Terminal Detail Status & Control

Remote Terminal can be reset from the EMS

The main tab provides an overview of current status

of RT components

Page 60: 5 Troubleshooting & Maint

61 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Remote Terminal Detail Status – Air Interface

Remote Terminal Modulation can be

switched (AM)

Primary & Backup HT signal Strength

Downlink RSSI and SQI

Tx Power

Current Modulation

Page 61: 5 Troubleshooting & Maint

62 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Operational TroubleshootingAir Interface

Page 62: 5 Troubleshooting & Maint

63 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Air Interface Troubleshooting

Use the HT Air Interface Statistics screen to look at UPLINK errors

Use RT Air Interface Statistics screen to look at DOWNLINK errors

Is the problem isolated to a single RT?– If yes, then concentrate on the RT side

Is the problem on some or all RTs in the sector?– If yes, then look at the HT side

Page 63: 5 Troubleshooting & Maint

64 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Air Link Statistics – HT (Uplink)Solicited Mode

= Real TimeCurrent Recording

Interval

This RT has pre-FEC uplink errors – OK?

Maybe...

...but it also has post-FEC errors -

BAD

Is SQI ok for the selected

Modulation?

Page 64: 5 Troubleshooting & Maint

65 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Air Link Statistics – RT (Downlink)Solicited Mode

= Real TimeCurrent Recording

Interval

This RT has pre-FEC downlink errors –

OK? Maybe...

...but it also has post-FEC errors -

BAD

Is SQI ok for the selected

Modulation?

Page 65: 5 Troubleshooting & Maint

66 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Air Interface Troubleshooting

Pre-FEC errors are ok as long as they are corrected– “Bit Error Count” shows pre-FEC errors

Is the link (UPLINK or DOWNLINK) experiencing post-FEC errors?– “Sync Errored Frames” and “FEC Errored Frames”

MUST be zero– Look at SQI (see next slide)– Look at Rx & Tx Levels – Do they match the planning

data?

Page 66: 5 Troubleshooting & Maint

67 Troubleshooting and Maintenance HUGHES PROPRIETARY II

What is SQI? SQI is Signal Quality Indicator The value of SQI is derived from the modem. It is a

measure of the received signal with respect to the “ideal” or perfect received signal.

Minimum acceptable SQI to get acceptable BER is about:– 12 dB for QPSK– 18 dB for 16QAM– 22 dB for 64-QAM

In clear sky, it’s possible to see SQI values at the RTs (downlink) in the mid 20's or more because the HT is transmitting at high power. – RTs close to the HT will see a stronger signal and will have

better SQI than RTs very far from the HT.   In the uplink, the SQI values (on the HT) won't be as high

as the downlink SQIs because RTs only transmit at a level high enough to meet the configured target RSSI level.  However, uplink SQI should still be close to or higher than the baseline numbers above.

Page 67: 5 Troubleshooting & Maint

68 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Operational TroubleshootingPorts and Services

Page 68: 5 Troubleshooting & Maint

69 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Port States & Status

Check the HT and RT Configuration screens to make sure ports that have services configured are in “Unlocked” state.

“Unlocked” means that alarms are reported to the EMS.

Remember, by default, all ports are created as “Locked for test” – they carry traffic but DO NOT report alarms

Two kinds of ports:– Physical : E1, DS3, STM-1– Virtual : IMA, HDLC, PPP

Page 69: 5 Troubleshooting & Maint

70 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Hub Terminal Detail Status – DS3 Port

STM-1 Port is similar. Look at Alarm bits on the interface.

Status of port: Configured vs. Operational

Is the port Unlocked?

Is the port failed?

Check Alarm bits

Problem on near end or far end?

Page 70: 5 Troubleshooting & Maint

71 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Remote Terminal Detail Status – E1 Port

Status of each port: Configured vs. Operational

Is the port failed?

Check E1 Alarm bits

Problem on near end or far end?

Loopback status

Page 71: 5 Troubleshooting & Maint

72 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Remote Terminal Detail Status – LAN Port

This port is Locked for Test – there will be no alarms reported to the

EMS

Does the speed and duplex match?

Page 72: 5 Troubleshooting & Maint

73 Troubleshooting and Maintenance HUGHES PROPRIETARY II

RT IMA Group Status

Don’t forget to check “virtual ports” like IMA groups

IMA Group State & Status

Individual E1 status

Page 73: 5 Troubleshooting & Maint

74 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Service States & Status

Check the Service Configuration screens (or service list screens) to make sure services are in “Unlocked” state.

Service states are on top of Port states

“Unlocked” means that alarms are reported to the EMS for that service

Page 74: 5 Troubleshooting & Maint

75 Troubleshooting and Maintenance HUGHES PROPRIETARY II

CES Service Detail Status

If no port number is given – than all circuits configured for the corresponding SSI

will be shown

HT Side of the service.

RT Side of the service.

Page 75: 5 Troubleshooting & Maint

76 Troubleshooting and Maintenance HUGHES PROPRIETARY II

LAN Service Detail StatusSelect HT side or RT side if

desired. Default is from both sides.

This service is “Unlocked” but Control Status

shows “Locked for Test” which means LAN port is “Locked

for Test”

Availability State is FAILED – LAN cable

unplugged? We should have already caught that before

coming here!

Page 76: 5 Troubleshooting & Maint

77 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Loopbacks

To help isolate a traffic problem it is possible to perform various loopbacks from the EMS. Note - Enabling any of the loopbacks will affect all services. Any changes made on the EMS database require a short time to download to the site. Loopbacks may require a maintenance window to perform.

Do not to use combinations of loopbacks simultaneously.

Page 77: 5 Troubleshooting & Maint

78 Troubleshooting and Maintenance HUGHES PROPRIETARY II

External Port Loopbacks

External port loopbacks are useful to verify physical connectivity

The administrative state of the port must be “Locked for test” to enable a loopback.

External port loopbacks can be applied to the HT or the RT.

Network HT RT

CPE

Traffic

Network HT RT

CPE

Traffic

RT Port External

Loop

HT Port External Loop

Page 78: 5 Troubleshooting & Maint

79 Troubleshooting and Maintenance HUGHES PROPRIETARY II

External Port Loopbacks – E1

If the port supports it, payload loopback can be used to isolate a problem

First start with External Line and then try External Payload

IDU

Tx

Rx

IDU

Framer

Rx

Tx

External line loopback loops the incoming data at the line

interface unit.

Set the BERT to internal clock.

External payload loopback loops the data at the framer

(Structured mode only).

Set the BERT to recovered clock.

Page 79: 5 Troubleshooting & Maint

80 Troubleshooting and Maintenance HUGHES PROPRIETARY II

External Port Loopbacks – E1 at HT

IDU

Tx

Rx

External Line Loopback – clocking is provided by the test equipment connected to that port.

Page 80: 5 Troubleshooting & Maint

81 Troubleshooting and Maintenance HUGHES PROPRIETARY II

External Port Loopbacks – E1 at HT

IDU

Framer

Rx

Tx

External Payload Loopback – clocking is provided by the system. Note that Payload

loopback is only available when the port is in Structured mode

Page 81: 5 Troubleshooting & Maint

82 Troubleshooting and Maintenance HUGHES PROPRIETARY II

External Port Loopbacks – STM1 at HT

IDU

Tx

Rx

IDU

Tx

Rx

Make sure that clocking settings are coordinated with the

test equipment settings.

Page 82: 5 Troubleshooting & Maint

83 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Internal Port Loopback

Internal loopback offers the ability to test the quality and integrity of a round trip service connection within the AB9000 system.

The loopback can be done at the HT or the RT. The administrative state of the service must be

“locked for test” to enable a loopback.

Network HT RT

CPE

Traffic

Network HT RT

CPE

Traffic

RT Port Internal Loop

HT Port Internal Loop

Page 83: 5 Troubleshooting & Maint

84 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Internal Port Loopback – E1 at RT

IDU

Tx

Rx

Internal line loopback at RT port is sending data back to HT.

Set the BERT to recovered clock.

Page 84: 5 Troubleshooting & Maint

85 Troubleshooting and Maintenance HUGHES PROPRIETARY II

OAM Test

Use this screen to perform Operations, Administration, and Management (OAM) testing on a selected ELCI to verify the integrity of the connection without affecting traffic on the circuit.

You can click to Start or Stop the OAM testingon the selected ELCI

Page 85: 5 Troubleshooting & Maint

86 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Service Troubleshooting

If the problem is related to a service, make sure that the configured services on the AIReach system match the connected equipment

For example:– Are the VP/VC’s the same (ATM)?– Are the bandwidths assigned the same (ATM, TDM,

LAN)?– Do MAC address or VLAN ID match (LAN)?– Do the port physical characteristics match?

• Framing, coding, etc (TDM)

• Scramble mode (ATM)

• Duplex setting (LAN)

Page 86: 5 Troubleshooting & Maint

87 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Is Site Visit Required? - Prerequisites

Questions to ask before going to site:

Can you ping HT or RT from EMS? Can you ping router at hub site from EMS? Can you ping the HT Air Interface IP address? Did I write down the IFL Cable loss values from HT

or RT Configuration screens? What type of unit is installed on the site?

– Do I have the correct spare IDU and ODU?– For example: 16 E1+STM-1+LAN + 26GHz B-Band

Page 87: 5 Troubleshooting & Maint

88 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Long Term Performance Monitoring

Page 88: 5 Troubleshooting & Maint

89 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Status and Performance Screens

Operators should baseline the site status and RF performance data of Hub and Remote sites during both clear sky and severe weather conditions.

They should perform a regular review of the Hub and Remote Air Link performance and check that RF performance is consistent.

Periodic monitoring of RSSI levels, Tx power, SQI, pre-FEC & post-FEC errors may help to identify problems as they develop, and before they become service affecting.

Page 89: 5 Troubleshooting & Maint

90 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Statistics

Operators should review HT and RT Air Link statistics and Service statistics for each site’s traffic bearing service type.

The historic log files can be reviewed by the operator or planner looking for unusual or intermittent unexplained RF and service anomalies that may have occurred during the past.

Statistics can be exported into Excel and plotted showing trends.

Page 90: 5 Troubleshooting & Maint

91 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Support Contact

Page 91: 5 Troubleshooting & Maint

92 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Contact Numbers Contacting TAC:

Phone: +1 301 601 7499

E-mail: [email protected]

Website https://customergateway.hns.com/

(Note: Access to customer‑specific and

product information is restricted by User ID and PIN.)

Mail: Hughes Network Systems, LLCTMD TAC 11717 Exploration LaneGermantown, MD 20876

Page 92: 5 Troubleshooting & Maint

93 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Hours of Operation Normal US business days - Monday though Friday,

8 a.m. to 5 p.m. EST.

– TAC engineers are on duty at the Germantown facility.

– In the event incoming calls cannot be answered by the on duty engineer during normal business hours, calls will automatically forward to an answering service, and the on duty engineer(s) will be paged.

US observed holidays, weekends, and normal business days after 5p.m. to 8a.m. EST.

– An automated attendant answers the TAC phone, and through menu selection, a customer can connect directly to the mobile phone of an on-call engineer.

Page 93: 5 Troubleshooting & Maint

94 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Hours of Operation (cont.)

– In the event the engineer’s mobile phone is not available, the caller can page the engineer through the answering service.

– During special customer situations outside of normal hours, such as, a planned major software load or system expansion, special assistance may be requested. To schedule this support contact the Director of the TAC at least one week in advance of the planned event.

– Every attempt is made to provide an adequate level of staffing during periods of inclement weather in Germantown, MD.

Page 94: 5 Troubleshooting & Maint

95 Troubleshooting and Maintenance HUGHES PROPRIETARY II

CRM Web Interface

Customer Resource Management (CRM) web-based interface

Customer portal for:– Opening trouble tickets (TAC Cases)– Viewing & Searching TAC Cases– Viewing documentation– Downloading software

Page 95: 5 Troubleshooting & Maint

96 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Login Page

Sign in

You must get a username and password from Hughes TAC

Page 96: 5 Troubleshooting & Maint

97 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Signed On

Site Navigation

Access Case history, Documents and other functions

here.

Click on Documents and Technical Info

Page 97: 5 Troubleshooting & Maint

98 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Documents By Category1. Click on Reports & Tech Info 2. Click on All

Documents with Folders

Here you will see a list of all document categories available

to you.

Click on a folder to open it and

navigate down

Page 98: 5 Troubleshooting & Maint

99 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – All Available Documents

1. Click on All Documents with Folders

Here you will see a complete list of

documents available.

You can enter a search term to quickly find a document.

This is a full text search so it

searches inside documents too.

Page 99: 5 Troubleshooting & Maint

100 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Report a New Case1. Click on BCN Case

Management

2. Click on Report a

Case

3. Select the Site Name. Hughes

typically does not track your site names, so you may see only

one entry here.

4. Fill in all of the required

information and provide as

much detail as possible

5. Attach screen

captures, database

dumps, logs, etc.

6. Submit the case

Page 100: 5 Troubleshooting & Maint

101 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Report a New Case

A new case is created and TAC is

notified.

Your case number is shown here.

Page 101: 5 Troubleshooting & Maint

102 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Search for Cases

1. Click on Case Search

2. Select a site. You can

select ALL SITES if you

are not sure of the site.

3. Other information is

optional

4. Click Search

Page 102: 5 Troubleshooting & Maint

103 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Search Results

Click on the case number to view the

case

Page 103: 5 Troubleshooting & Maint

104 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Case Overview

Case status is displayed here

You can add notes and

attachments to the case anytime

Hughes TAC updates are

shown here as well

Click here to show the case

description details

Page 104: 5 Troubleshooting & Maint

105 Troubleshooting and Maintenance HUGHES PROPRIETARY II

TAC Online – Case Details

Case description

details

Page 105: 5 Troubleshooting & Maint

106 Troubleshooting and Maintenance HUGHES PROPRIETARY II

Conclusion

From this course you should:– Understand the distinction between:

• Installation Troubleshooting• Operational Troubleshooting

– Know how to use the LCT to troubleshoot– Know how to use the EMS to troubleshoot

– Understand how to contact Hughes TAC for issues that you can’t resolve

Questions?