nicmos suspend event stb 632 “ mech_2_max_retries_exceeded” 2008/069
DESCRIPTION
NICMOS Suspend Event STB 632 “ MECH_2_MAX_RETRIES_EXCEEDED” 2008/069. NICMOS STB 632 “ MECH_2_MAX_RETRIES_EXCEEDED”. At 2008/069/02:46:03 UTC, NICMOS Suspended with an NICMOS 632 status buffer message (parameter = 225 octal, 149 decimal) - PowerPoint PPT PresentationTRANSCRIPT
NICMOS Suspend EventSTB 632
“MECH_2_MAX_RETRIES_EXCEEDED” 2008/069
2
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• At 2008/069/02:46:03 UTC, NICMOS Suspended with an NICMOS 632 status buffer message (parameter = 225 octal, 149 decimal)
• Indicates Out of Tolerance move for a commanded move of Filter Wheel 2 (FW2) to the F110W filter position– Tolerance is 110 counts
• STB parameter indicates a disparity of 149 counts
– FSW analysis indicates that the final FW2 position was only 13 counts away from the expected, indicating FW2 did settle at the correct position
• NICMOS was in Observe executing proposal 11168– The IMF in the Hidden Galactic Starburst W49A
3
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• 10 prior FW 2 moves had been successful since transitioning from SAA Operate to Operate– 069/02:12:41 – Transition from SAA Operate to Operate complete
– 069/02:14:28 - 02:43:38 – Blank, F110W, F165M, F207M, F110W, F165M, F207M, F110W, F165M, F207M
– 069/02:45:50 – F110W (NICMOS 632)
• At the time of the event:– HST was not in an SAA passage
– HST had entered orbit day at ~069/02:16 (roughly orbit noon)
• HSTAR 11215 has been opened to document this event
4
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• Original occurrence in 1998 – HSTAR 6815– Occurred during a move from Blank to F110W– Attributed to known noise in the FW2 electronics
• Most recently in 2007/158– HSTAR 10851– Occurred during a move from F171M to F110W– Also attributed to known noise in the FW2 electronics
• From HSTAR 10851 closure: “Ground testing did demonstrate this susceptibility, however, and it is attributed to noise in the electronics. Bench testing of these mechanisms before integration into NICMOS showed torsional oscillation took ~8 seconds to completely dampen out after a motion was commanded. Review of the code revealed that a settling time of 3 seconds is allocated between moves, but not between moves and position measurements. Therefore the Filter Wheel is still ringing when the FSW is reading out the inductosyns. This will occasionally results in an erroneous measurement which is outside the miss-distance tolerance of 110 counts.”
– SCR N52 - NICMOS Filter Wheel retry failures opened 7/19/07
• Single retry events remain common on all three Filter Wheels
5
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 632
6
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• OR 18212-0 to dump NICMOS memory per ROP NS-09 was completed at ~069/05:09– Dump files transferred to Payloads FSW for analysis
• Engineering telemetry extracted and examined for any anomalous behavior with FW2 or the Mechanism converter (see following)– Nominal FW2 move from 068 included for comparison
7
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend EventFilter Wheel 2 Temperature
069/ 02:31:00 - 02:51:00
-10
-9
-8
-7
-6
-5
-4
-3
-2
-1
0
2008
.069
02:
35:0
7.49
1
2008
.069
02:
36:3
7.49
1
2008
.069
02:
37:0
8.49
1
2008
.069
02:
37:3
9.49
1
2008
.069
02:
38:2
8.99
1
2008
.069
02:
39:0
7.99
1
2008
.069
02:
39:3
8.49
1
2008
.069
02:
40:0
9.99
1
2008
.069
02:
41:0
7.49
1
2008
.069
02:
41:3
7.99
1
2008
.069
02:
42:0
9.49
1
2008
.069
02:
42:3
9.99
1
2008
.069
02:
43:3
7.49
1
2008
.069
02:
44:0
8.49
1
2008
.069
02:
44:3
9.49
1
2008
.069
02:
45:2
8.99
1
2008
.069
02:
46:0
7.99
1
2008
.069
02:
46:3
8.49
1
2008
.069
02:
47:0
9.99
1
2008
.069
02:
48:0
7.49
1
2008
.069
02:
48:3
7.99
1
2008
.069
02:
49:0
9.49
1
2008
.069
02:
50:2
8.99
1
Time (UTC)
Tem
per
atu
re (
deg
C)
NFW2TMP
8
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend EventFilter Wheel 2 Temperature
068/ 21:27:00 - 21:47:00
-10
-9
-8
-7
-6
-5
-4
-3
-2
-1
0
2008
.068
21:
27:3
7.49
0
2008
.068
21:
28:0
9.99
0
2008
.068
21:
29:0
8.49
0
2008
.068
21:
30:0
7.49
0
2008
.068
21:
30:3
9.49
0
2008
.068
21:
31:3
7.99
0
2008
.068
21:
32:2
8.99
0
2008
.068
21:
33:0
9.49
0
2008
.068
21:
34:0
7.99
0
2008
.068
21:
34:3
9.99
0
2008
.068
21:
35:3
8.49
0
2008
.068
21:
36:3
7.49
0
2008
.068
21:
37:0
9.99
0
2008
.068
21:
38:0
8.49
0
2008
.068
21:
39:0
7.49
0
2008
.068
21:
39:3
9.49
0
2008
.068
21:
40:3
7.99
0
2008
.068
21:
41:2
8.99
0
2008
.068
21:
42:0
9.49
0
2008
.068
21:
43:3
7.99
0
2008
.068
21:
44:2
8.99
0
2008
.068
21:
45:0
9.49
0
2008
.068
21:
46:0
7.99
0
2008
.068
21:
46:3
9.99
0
Time (UTC)
Tem
per
atu
re (
deg
C)
NFW2TMP
9
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event+/- 12 V Resolver Voltages; +15 V, +5 V Mechanism Voltages
069/ 02:31:00 - 02:51:00
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
14
1620
08.0
69 0
2:35
:07.
491
2008
.069
02:
36:3
7.49
1
2008
.069
02:
37:0
8.49
1
2008
.069
02:
37:3
9.49
1
2008
.069
02:
38:2
8.99
1
2008
.069
02:
39:0
7.99
1
2008
.069
02:
39:3
8.49
1
2008
.069
02:
40:0
9.99
1
2008
.069
02:
41:0
7.49
1
2008
.069
02:
41:3
7.99
1
2008
.069
02:
42:0
9.49
1
2008
.069
02:
42:3
9.99
1
2008
.069
02:
43:3
7.49
1
2008
.069
02:
44:0
8.49
1
2008
.069
02:
44:3
9.49
1
2008
.069
02:
45:2
8.99
1
2008
.069
02:
46:0
7.99
1
2008
.069
02:
46:3
8.49
1
2008
.069
02:
47:0
9.99
1
2008
.069
02:
48:0
7.49
1
2008
.069
02:
48:3
7.99
1
2008
.069
02:
49:0
9.49
1
2008
.069
02:
50:2
8.99
1
Time (UTC)
Vo
ltag
e (V
)
N12V1RES
N12V2RES
N15VMECH
N5VMECH
NM12V1RS
NM12V2RS
10
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend Event+/- 12 V Resolver Voltages; +15 V, +5 V Mechanism Voltages
068/ 21:27:00 - 21:47:00
-14
-12
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
14
1620
08.0
68 2
1:27
:37.
490
2008
.068
21:
28:0
9.99
0
2008
.068
21:
29:0
8.49
0
2008
.068
21:
30:0
7.49
0
2008
.068
21:
30:3
9.49
0
2008
.068
21:
31:3
7.99
0
2008
.068
21:
32:2
8.99
0
2008
.068
21:
33:0
9.49
0
2008
.068
21:
34:0
7.99
0
2008
.068
21:
34:3
9.99
0
2008
.068
21:
35:3
8.49
0
2008
.068
21:
36:3
7.49
0
2008
.068
21:
37:0
9.99
0
2008
.068
21:
38:0
8.49
0
2008
.068
21:
39:0
7.49
0
2008
.068
21:
39:3
9.49
0
2008
.068
21:
40:3
7.99
0
2008
.068
21:
41:2
8.99
0
2008
.068
21:
42:0
9.49
0
2008
.068
21:
43:3
7.99
0
2008
.068
21:
44:2
8.99
0
2008
.068
21:
45:0
9.49
0
2008
.068
21:
46:0
7.99
0
2008
.068
21:
46:3
9.99
0
Time (UTC)
Vo
ltag
e (V
)
N12V1RES
N12V2RES
N15VMECH
N5VMECH
NM12V1RS
NM12V2RS
11
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend EventMechanism Current
069/ 02:31:00 - 02:51:00
-0.010
-0.009
-0.008
-0.007
-0.006
-0.005
-0.004
-0.003
-0.002
-0.001
0.000
2008
.069
02:
35:0
7.49
1
2008
.069
02:
36:3
7.49
1
2008
.069
02:
37:0
8.49
1
2008
.069
02:
37:3
9.49
1
2008
.069
02:
38:2
8.99
1
2008
.069
02:
39:0
7.99
1
2008
.069
02:
39:3
8.49
1
2008
.069
02:
40:0
9.99
1
2008
.069
02:
41:0
7.49
1
2008
.069
02:
41:3
7.99
1
2008
.069
02:
42:0
9.49
1
2008
.069
02:
42:3
9.99
1
2008
.069
02:
43:3
7.49
1
2008
.069
02:
44:0
8.49
1
2008
.069
02:
44:3
9.49
1
2008
.069
02:
45:2
8.99
1
2008
.069
02:
46:0
7.99
1
2008
.069
02:
46:3
8.49
1
2008
.069
02:
47:0
9.99
1
2008
.069
02:
48:0
7.49
1
2008
.069
02:
48:3
7.99
1
2008
.069
02:
49:0
9.49
1
2008
.069
02:
50:2
8.99
1
Time (UTC)
Cu
rren
t (A
)
NMECHCUR
12
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS 2008/069 Suspend EventMechanism Current
068/ 21:27:00 - 21:47:00
-0.01
-0.009
-0.008
-0.007
-0.006
-0.005
-0.004
-0.003
-0.002
-0.001
0
2008
.068
21:
27:3
7.49
0
2008
.068
21:
28:0
9.99
0
2008
.068
21:
29:0
8.49
0
2008
.068
21:
30:0
7.49
0
2008
.068
21:
30:3
9.49
0
2008
.068
21:
31:3
7.99
0
2008
.068
21:
32:2
8.99
0
2008
.068
21:
33:0
9.49
0
2008
.068
21:
34:0
7.99
0
2008
.068
21:
34:3
9.99
0
2008
.068
21:
35:3
8.49
0
2008
.068
21:
36:3
7.49
0
2008
.068
21:
37:0
9.99
0
2008
.068
21:
38:0
8.49
0
2008
.068
21:
39:0
7.49
0
2008
.068
21:
39:3
9.49
0
2008
.068
21:
40:3
7.99
0
2008
.068
21:
41:2
8.99
0
2008
.068
21:
42:0
9.49
0
2008
.068
21:
43:3
7.99
0
2008
.068
21:
44:2
8.99
0
2008
.068
21:
45:0
9.49
0
2008
.068
21:
46:0
7.99
0
2008
.068
21:
46:3
9.99
0
Time (UTC)
Cu
rren
t (A
)
NMECHCUR
13
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• Memory dumps have been analyzed by Payloads FSW– EEPROM shows the expected 5-byte difference for the stored motor
commutations– No out of limit events recorded in internal limit table– Internal error log has several instances of previous motor retries
• Error Number 633 = FW2 Retry (internal ESB only)– 19 events since 2008/001
• Error Number 632 = FW2 Retries Allowed Exceeded (Suspend)
---------------------------------------------------------------------------- 54 | 2008/069/00:54:13.803 | 633 | 121 | 0x0075---------------------------------------------------------------------------- 55 | 2008/069/01:00:33.804 | 633 | 121 | 0x0071 ---------------------------------------------------------------------------- 56 | 2008/069/02:45:56.791 | 633 | 121 | 0x0072 ---------------------------------------------------------------------------- 57 | 2008/069/02:46:02.894 | 632 | 120 | 0x0095 ----------------------------------------------------------------------------
14
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• Forward Plan– All NICMOS telemetry indicates there are no hardware or operational
concerns
• Filter Wheel retries are commonplace occurrences
– Recommend recovery to Operate and repositioning of the PAM using real time commanding
• See MEGG following for command windows
– Next external NICMOS science observation is scheduled for 069/16:15:16
• Proposal 11157 - NICMOS Imaging Survey of Dusty DebrisAround Nearby Stars Across the Stellar Mass Spectrum
15
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• Operations Request 18213-0– Recover NICMOS from Suspend to Hold mode
• Operate converter relay opened on Side 1• Suspend flag and Checksum error flag cleared• NICMOS RTCS processing enabled
– Transition NICMOS from Hold to Boot• Operate converter relay closed on Side 1• Suspend AP enabled• Boot FSW initialized
– Transition NICMOS from Boot to SAA Operate• Operate FSW copied to RAM and initialized• Buffer Boxes turned on• Mechanisms initialized• PAM sensor turned on
– Transition NICMOS from SAA Operate to Operate• Initialize and enable detectors
– Wait at least 1 hour from SAA Operate before moving PAM– Move PAM to Camera 2 coronagraphic position (PAMC) in preparation for
intercepting the science timeline
16
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS Recovery to “Operate”
17
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS Recovery to “Operate” cont’d PAM Move (At least 1 hr after NICMOS to SAA Operate)
18
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS First Science Observations (Proposal 11157)
19
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
NICMOS First Science Observations (Proposal 11157)
20
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• On-orbit Validation and Monitoring– SI SE will monitor transition to SAA Operate in the SMOR
– SI SE will monitor results of first observation and filter wheel move remotely (actual first observation occurs during LOS)
• System Impacts– SI
• NICMOS ready for science
– DMS, EPS, I&C, OTA, PCS, Safing, TCS
• No impact
– Critical Science Considerations
• None
21
NICMOS STB 632 “MECH_2_MAX_RETRIES_EXCEEDED”
• Follow-up Activities