glast large area telescope lat pre-shipment review ncrs and waivers pat hascall

NCRs and Waivers 1

GLAST LAT Project September 15, 2006: Pre-Shipment Review

Presentation 9 of 12

GLAST Large Area TelescopeGLAST Large Area Telescope

LAT Pre-Shipment Review

NCRs and Waivers

Pat HascallSystems Engineering

Stanford Linear Accelerator Center

Gamma-ray Large Gamma-ray Large Area Space Area Space TelescopeTelescope

NCRs and Waivers 2



NCR Introduction NCR Introduction

• Presentation focus is on the main hardware related NCRs that remain open– Impact assessment for those NCRs identified as “Can Not

Duplicate” (CND) will be discussed.– Several NCRs were left open for Environmental Testing at

NRL and will be closed when final data review is completed after TV (Documentation of analysis or minor rework/inspection)

• NCRs are classified into categories for discussion purposes

• NCR Summary List of all open NCRs is presented for reference

NCRs and Waivers 3



NCR Category DefinitionsNCR Category Definitions

Open NCR’s classified into categories for discussion purposes:

Category Definition Count

Hardware Discrepancy H/W Issue, half can be closed with minor rework or inspection after TV

6

FSW Discrepancy Identified FSW bug, FSW JIRA in work or completed 6

Monitor for Verification Likely test issue, trending for repeat 2

Known Feature Specification not violated, but trending or changes required to accommodate behavior

11

EGSE/Data Processing NCR has been isolated to EGSE/Data Processing 12

CND Could not duplicate 7

NCRs and Waivers 4



NCR Resolution DefinitionsNCR Resolution Definitions

Open NCR’s classified into categories for discussion purposes:

Closure Plan

Definition Count

Issue resolved, closure plan in place or no impact to LAT

Closing Issue resolved, documentation required to close 13

Close after TV Monitored during TV with no issues seen (complete writeup and close) or minor rework scheduled after TV

5

B0.6.11 Resolved with FSW release B0.6.11 1



Closure deferred

EGSE issues, low priority to fix or hold open to track repeats

9

Work continuing

QAR CND or other issue expected to be transferred to a QAR 7

In process Investigation ongoing 3

Waiver Waiver in process 1

NCRs and Waivers 5



NCR Resolution vs CategoryNCR Resolution vs Category

Category B0.

6.11

B0.

7.0

B1.

0.0

Clo

se a

fter

TV

Clo

sure

def

erre

d

Clo

sing

in p

roce

ss

QA

R

Wai

ver

TotalEGSE 8 4 12FSW 1 1 1 3 6Hardware Discrepancy 1 1 3 1 6Known Feature 2 2 1 6 11Monitor for verification 2 2CND 7 7

Total 1 4 1 5 9 13 3 7 1 44

Resolution

NCRs and Waivers 6



NCRs For DiscussionNCRs For Discussion

NCR Date Opened

Affected Hardware

Problem Description NCR Category

Resolution

855 3/23/06 DAQ LATC verify occcasionally reports errors in CRC, SPT, ARC

CND QAR

880 4/12/06 DAQ SIU Spontaneous Reboot CND QAR

881 4/12/06 DAQ EPU Spontaneous Reboot CND QAR

902 5/9/06 DAQ EPU 0 unexpected reboot CND QAR

922 6/5/06 DAQ Housekeeping telemetry stopped FSW Discrepancy

in process

946 7/11/06 ACD ACD FREE board 5 not responsive after power up

CND QAR

948 7/19/06 DAQ EPU 1 reboot, EPU 2 reboot CND QAR

949 7/21/06 DAQ Primary SIU reboot, redundant SIU reboot

CND QAR

957 8/4/06 DAQ EPU 0 CPU junction temperature higher than expected in vacuum

Hardware Discrepancy

Closure near

975 9/1/06 DAQ Datagram timetag error FSW Discrepancy

in process

NCRs and Waivers 7



NCR 855NCR 855LATC Verify ErrorsLATC Verify Errors

• Issue– Calorimeter and Tracker front-end register (RC and FE) was not read successfully

• Analysis– Issue in 16 of 3805 LATC configurations

• Affects about 10 bits of ~ 2 million bits written/read in each LATC execution– Mostly at the start of commissioning:

• Fourteen happened in first 290 runs • One in run 777 • One in run 1426• None in last 2400 write/reads • None using FSW B0.6.9 (2120 runs)

– Have not been able to replicate on the testbed

• Resolution Plan– Mitigation plans in FSW

• JIRA FSW-653 (B0.6.10) adds a reset to the calorimeter at power up• JIRA FSW-729 adds a retry by LATC when a verify error occurs, to be implemented

if the problem reappears• Impacts on On-orbit performance

– With no action, the LAT will not start the physics acquisition and one orbit worth of data would not be collected

CND, QARCND, QAR

NCRs and Waivers 8



NCR 922NCR 922Housekeeping Telemetry StoppedHousekeeping Telemetry Stopped

• Issue:– During a charge injection test, LAT housekeeping process

(LHK) has twice emitted an error and subsequently did not create telemetry packets successfully

• Analysis:– Review by FSW indicates a possible cause that the LHK and

LCI (LAT Charge Injection) process may be in contention for memory

– Was reproduced on testbed, analysis in progress• Resolution Plan:

– Find and correct FSW error• Impacts on On-Orbit performance:

– Loss of housekeeping data until restart

QARQAR

NCRs and Waivers 9



NCR 946NCR 946ACD Free 5 Power UpACD Free 5 Power Up

• Issue:– ACD Free Board 5 was not responsive to commands after power up

• Analysis:– Other power up parameters (voltages and currents) reviewed with no

abnormalities seen– Occurred once in over 275 power ups (122 using redundant GASU)– Two possible causes

• Look-at-me from redundant GASU did not reach or might have been decoded improperly on the Free board in which case it would only respond to commands from the primary (unpowered) GASU

• Custom power up sequence (compensates for an old bug in ASIC primary/redundant switching code) did not result in successful power up in redundant mode

• Resolution Plan:– Cycle power to FREE card (interim solution)– Add functionality to FSW to be able to send Look-at-me command to a FREE

card without power cycling (JIRA FSW 718 adds capability, targeted for B1.0.0)

• Impacts on On-Orbit performance:– Extend power up sequence timeline slightly

CND, QARCND, QAR

NCRs and Waivers 10



CPU RebootsCPU Reboots

• Issue– EPU and SIU reboots seen on 7 occasions (next chart has details)

• Analysis– Several of these have likely causes relating to EGSE – Two recent reboots have occurred without the EGSE symptoms– Rate is 7 in about 1800 hours, or once per 250 hours– Existing data does not give sufficient information to determine the

cause• Resolution Plan

– FSW diagnostic software to be loaded that will help diagnose the cause of the reboots by identifying which tasks are executing

• Records most recent 1024 task switches• Can be read out in primary or secondary boot

– Run LAT on a non-interference basis at observatory to accumulate run hours

• Impacts on On-orbit performance– Loss of data and LAT housekeeping telemetry until SIU is rebooted,

potential for SC directed loadshed– Loss of ½ the science data until EPU is rebooted

CND, QARCND, QAR

NCRs and Waivers 11



CPU Reboots (Continued)CPU Reboots (Continued)

NCR Unit Type of reboot LAT Activity Potential Cause

880 SIU redundant

VxWorks reboot TkrTotGain_SVC_500hz

881 EPU2 uncorrectable memory error

TkrTotGain_SVC_500hz Transient memory error

902 EPU2 CPU exception, PPC Vector 0x300 (DSI)

During LatReinit, concurrent with main feed on command

EPU2 was in primary boot and transmitting boot telemetry when script rebooted the SIU. This is a nonstandard configuration that may have contributed to the reboot

948-1 EPU1 Watchdog LPA: tackscan-6_0.55hr LCB errors at the time of the reboot indicate that the VSC was falling behind.

949-1 SIU primary

Watchdog e2e_LAT_22xGammafilterNoPer_0.17hr

1 pps/timetone errors from all 3 processors, and timehack table entry errors from the SIU indicate reboot likely induced due to incorrect sequence 1 PPS and timetone messages from the VSC

948-2 EPU2 Checkstop LAT-22x_0.50hr muon run

949-2 SIU redundant

Watchdog intSeAppLrs_e2e_LAT-22xGammafilterNoPer_0.50hr

No clear pattern has developedNo clear pattern has developed

NCRs and Waivers 12



NCR 957NCR 957EPU 0 CPU Junction TemperatureEPU 0 CPU Junction Temperature

• Issue:– After applying vacuum, EPU 0 CPU junction temperature was higher than the other

CPUs by up to 28 degrees and reached 104 degrees during bakeout. Telemetry returned to normal after repressurization.

• Analysis:– Not observed at unit test or ambient level LAT testing– Possible causes

• Degradation of thermal connection from heat sink to CPU • CPU on-chip sensor defective (unlikely)

– Thermal analysis of board and tests on similar hardware support first hypothesis (disconnected heat sinkn test and analysis results consistent with observed temperature rise)

– Under this condition, thermal and reliability margin still exceeds requirements• Expected junction temperature for this box at hot is 80 degrees

– No potential for electrical side effects• Heatsink is brazed to the rail, and is adjacent to a empty slot• Epoxy is not electrically conductive• Board is conformal coated

• Resolution Plan:– Use as is

• Impacts on On-Orbit performance:– None

NCRs and Waivers 13



NCR 975NCR 975Datagram Time TagsDatagram Time Tags

• Issue:– Some datagrams have a time tag that is 4.2 seconds “fast”

• Analysis:– Rate is about 1 in 10000 datagrams– Likely FSW bug– Analysis in progress

• Will reproduce on testbed– Could potentially be worked around on the ground

• Resolution Plan:– Find bug and fix it

• Impacts on On-orbit performance– None, if fixed

• May use GSW mitigation until FSW error is found and fixed

NCRs and Waivers 14



Open NCRsOpen NCRs

NCR Date Opened

Affected Hardware


Resolution Could Not Duplicate

535 6/20/05 TKR Monitor trend data for TKR 4 thru LAT testing (spin-off NCR)

Known Feature

Close after TV

624 8/27/05 ACD Fluctuations in temperature reading (originally an ACD PR)

Monitor for Verification

Close after TV

625 8/27/05 ACD AcdVetoHitmapPha apparent retrigger Known Feature

Closure near

626 8/27/05 ACD ACDMonitor script high counts during T/V (originally ACD PR)

Monitor for Verification

Close after TV

684 10/3/05 TKR Noise Occupancy failures due to intermittent hot strips.

Known Feature

Closure near

718 10/27/05 ACD VETO threshold for channel 1123 can't be set below minimum value of 0.45pC (originally an ACD PR)

Known Feature

Close after TV

806 1/17/06 Trigger CAL is retriggering during SVAC run Known Feature

Closing

840 3/8/06 DAQ RunControl software problems observed on spare PDU during T/V test (NCR #794); opened NCR to track FSW updates

Known Feature

B0.7.0

851 3/16/06 DAQ One or more EPU resets unexpectedly due to large number of events and interaction with end of run activities

FSW Discrepancy

B1.0.0

852 3/16/06 DAQ LCB errors observed during muon run EGSE Closure deferred

855 3/23/06 FSW LATC verify occcasionally reports errors in CRC, SPT, ARC

CND QAR CND

NCRs and Waivers 15



Open NCRsOpen NCRs

NCR Date Opened

Affected Hardware



859 3/23/06 FSW CAL LCI data compression less than expected, results in very large datagrams

FSW Discrepancy

B0.6.11

880 4/12/06 Integration SIU Spontaneous Reboot CND QAR CND

881 4/12/06 Integration EPU Spontaneous Reboot CND QAR CND

882 4/12/06 DAQ GLAT2525 power up software crash due to Result FIFO not empty.

Known Feature

B0.7.0

884 4/15/06 Integration VSC errors observed during LAT power on EGSE Closure deferred

894 4/29/06 Integration LATC dump GTFE mask error FSW Discrepancy

in process

902 5/9/06 DAQ EPU 0 unexpected reboot CND QAR CND

909 5/16/06 EGSE LAT06X EEPROM writecount test results misinterpreted

EGSE Closing

913 5/23/06 DAQ FSW error in packing ACD charge injection data FSW Discrepancy

B0.7.0

922 6/5/06 DAQ Housekeeping telemetry stopped FSW Discrepancy

in process

932 6/20/06 TKR RS103 susceptiblity in tracker Hardware Discrepancy

Waiver

933 6/22/06 EGSE EGSE overwrites timetone messages EGSE Closure deferred

NCRs and Waivers 16



Open NCRsOpen NCRs

NCR Date Opened

Affected Hardware



936 6/24/06 EGSE EGSE timed out on the 1553 bus EGSE Closure deferred

938 6/26/06 EGSE Housekeeping packet sampled early, causing test error

EGSE Closure deferred

939 6/28/06 Thermal CS102 sensitivity in RTD sensors Hardware Discrepancy

B0.7.0

941 6/28/06 Mech Temperature sensors swapped Hardware Discrepancy

Closing

942 6/29/06 Mech ACT to LAT blanket interface not consistent Hardware Discrepancy

Close after TV

945 7/6/06 DAQ EGSE map of FSW files (in FMX) not properly initialized

EGSE Closing

946 7/11/06 ACD ACD FREE board 5 not responsive after power up CND QAR CND

948 7/19/06 DAQ EPU 1 reboot, EPU 2 reboot CND QAR CND

949 7/21/06 DAQ Primary SIU reboot, redundant SIU reboot CND QAR CND

957 8/4/06 DAQ EPU 0 CPU junction temperature higher than expected in vacuum

Hardware Discrepancy

Closing

958 8/7/06 DAQ GTRC phase error induced by FPGA bug Known Feature

Closure deferred

959 8/8/06 EGSE EGSE VSC crashes EGSE Closure deferred

960 8/8/06 EGSE Script missed an LPA stop command, resulting in test error


NCRs and Waivers 17



Open NCRsOpen NCRs

NCR Date Opened

Affected Hardware



966 8/14/06 EGSE EGSE current measurement scale incorrect EGSE Closing

967 8/14/06 EGSE EGSE temperature measurements inaccurate over temp


969 8/16/06 ACD ACD FREE HV monitors read 100V at cold Known Feature

Closing

971 8/18/06 TKR GTFE (0,-x6,20) readback intermittent at cold Known Feature

Closing

975 9/1/06 DAQ Datagram timetag error FSW Discrepancy

in process

976 9/2/06 DAQ Watchdog reboot at hot induces correctable memory errors

Known Feature

Closing

977 9/5/06 EGSE Script sampled telemetry too soon, causing test error EGSE Closing

NCRs and Waivers 18



LAT Waivers (1/3)LAT Waivers (1/3)

CCR # Title Description Status

433-0311 DC Voltage Tolerance

LAT is required to tolerate 0-40V DC. Due to MOSFET switches at power feed inputs, LAT can tolerate minimum 15V, excluding transient events. Approved

433-0356Test Point Short Circuit Isolation

LAT is required to operate within spec if any testpoint is shorted to ground. A shorted external clock select pin would render the redundant GASU inoperable. Approved

433-0357 DC Voltage Tolerance #2

LAT required to tolerate 0-40V DC. After a voltage drop analysis, it was found that the TEM MOSFET switches would receive too low a voltage with the DAQ feed voltage at 15V. To operate the TEM's safely, the input voltage needs to be 18.5V minimum. Approved

433-0358 GTFE TID

LAT is required to perform TID testing on all GTFE ASIC lots. The final two lots were not tested since previous lots exhibited such large margins. Approved

433-0360

Tracker Environmental Test With Non-Flt or Missing Cables

Several tracker towers went through environmental test with a subset of missing or non-flight flex cables. The replacement flight cables were not subjected to component-level vibe and will not see twelve tvac cycles. Approved

NCRs and Waivers 19





433-0361 24AWG STD Strength Cu

High strength Cu alloy is required for 24AWG wire. LAT uses standard strength Cu wire. As reported by the LAT PCB, standard strength 24AWG wire has been used on previous NASA projects with GSFC’s approval with no compromise to product reliability. Approved

433-0362 J-STD vs NASA STD

LAT circuit card assemblies uses J-STD-001 as the workmanship standard instead of NASA-STD-8739.3. Approved

433-0367Tracker Flex Cable and MCM Coupon Failures

Several flex cables and MCM’s are installed on the LAT although they have failed coupons. Approved

433-0368 Radiator Sine Vibe

The radiators will not be installed for LAT-level sine vibe test. Instead, the radiators were subjected to alternative tests, i.e. pull test, tap test, LAT-level acoustic test. Approved

433-0369 EMI Skirt Stay ClearCenter EMI skirt pieces near SC-LAT flexures exceed the LAT stay-clear by 0.015” max. Approved

433-0374 VCHP CECM

The VCHP feed violates the CECM requirement. The measured value is ~700mVp-p vs the requirement of 200mVp-p. Approved

NCRs and Waivers 20





446-0402Keyed Test Point Connectors Test point connectors are not keyed. Approved

446-0403Segregate Signal Types on Test Connectors JL-39 contains signals of different signal classes. Approved

446-0XXRS103 TKR Noise Occupancy

Tracker noise occupancy exceeded its limit of 10-4 during RS 103 (vertical polarization) between 95-101MHz.

Submitted to GSFC

446-0XX Cable Shield TerminationFloating shields are not properly terminated per NASA-STD- 8739.4.

Submitted to GSFC

NCRs and Waivers 21



SC-LAT ICD WaiversSC-LAT ICD Waivers

ICN # Title Description Status

-095LAT Grid Interface Hole Out-of-Tolerance

Several grid interface hole locations are out of tolerance. Using the as-built LAT Grid and SC interface hole locations, the analysis shows the predicted forces to align the shear pins are small and a minimum of 0.007” exist between the bolts and holes in the flexures and mating should not be an issue. Approved

-107Recessed Grid Bushings

The +Y and –Y LAT grid interface hole bushings are recessed by 0.022” worst case. Stress analysis at the SC mount interface shows the margins of safety for ultimate and yield bearing strength is 7% which is acceptable. The margin of safety for pin bending is >200%. Approved

NCRs and Waivers 22



NCR and Waiver Summary NCR and Waiver Summary

• The LAT Team is confident that none of the NCRs or Waivers presented are significant enough to prevent the LAT from shipping.

• The LAT Team recommends proceeding with the LAT shipping as planned.

glast large area telescope lat pre-shipment review ncrs and waivers pat hascall

Documents