embraer risk assessment “three flags”...
TRANSCRIPT
EMBRAER – Safety Review Board
Embraer Risk Assessment
“Three Flags” Method
Maximilian Kleinubing – B.S.
Flight Test Engineer
Embraer Gavião Peixoto
Email: [email protected]
EMBRAER – Safety Review Board
EMBRAER – Safety Review Board
Embraer Risk AssessmentEmbraer Risk Assessment
“Three Flags” Method
EMBRAER – Safety Review Board• Introduction
• Objective
• References
• Flight Envelope Analysis
• Test Point Execution Analysis
• System Failures Analysis• System Failures Analysis
• Risk Management
• Conclusions
EMBRAER – Safety Review Board
Introduction
Risk Assessment methodology has a key impact on safety during flight tests.
It is considered that the best methodology will be the one that is concise,
effective and has the ability to unveil the hazardseffective and has the ability to unveil the hazardsinvolved on the tests.
EMBRAER – Safety Review Board
Objective
• The goal of this presentation is to show how Embraer performs its Flight Test Risk how Embraer performs its Flight Test Risk Assessments at the present day.
EMBRAER – Safety Review Board
References
• Embraer ENS 00650
• FAA Order 4040.26• FAA Order 4040.26
EMBRAER – Safety Review Board
• Why Three Flags ?
EMBRAER – Safety Review Board
Flight EnvelopeTest Point
ExecutionSystem Failures
ACJ 1309 evaluation
EMBRAER FLIGHT TEST RISK ASSESSMENT METHOD
Operational Envelope
Design
Three Flags Method
ACJ 1309 evaluation
combined with flight
test effects to each
failure
Sequence of Assessment
Limited
Screen Factor Method
Each analysis is independent from each other.
EMBRAER – Safety Review Board
Flight Envelope
EMBRAER – Safety Review Board
Flight Envelope Analysis
FLIGHT ENVELOPE RISK CLASSIFICATION
OPERATIONAL ENVELOPE LOW
LIMITED ENVELOPE MEDIUMLIMITED ENVELOPE MEDIUM
DESIGN ENVELOPE MEDIUM
SCREEN FACTOR HIGH
EMBRAER – Safety Review Board
“Operational Envelope”: Envelope flown by operational
pilots. Normally is defined by protection systems and
placards.
“Design Envelope”: Envelope established by design
reports and larger than the operational envelope.
DEFINITIONS
reports and larger than the operational envelope.
“Limited Envelope”: Envelope that was limited during
the developmental flight testing due to engineering
problems encountered.
“Screen Factor”: Aims to detect if the a flight test is high
risk based on lack of sufficient experience, limited
engineering models and critical damages potential.
EMBRAER – Safety Review Board“Screen Factor” Application to High Risk Detection
THREE FLAGS
Screen Factor Tool
I – Detect Expertise on the Proposed Tests
Previous tests are sufficient to predict a safe behavior of the new proposed tests ?
II – Detect Limitations of Modeling Tools
Best available modeling tools are sufficient to predict a safe behavior for
the new proposed tests?
III – Detect Type of Possible Effects of the Proposed Test
Hazardous or catastrophic effects might result from the proposed tests if
predictions are incorrect?
EMBRAER – Safety Review BoardTest Point Execution
EMBRAER – Safety Review BoardTest Point Execution
• Considering that you might NOT have performed the maneuver ever before, the
pilot/FTE can count only with:
• Flight Experience
• Expertise on the aircraft
• Expertise on that type of maneuver• Expertise on that type of maneuver
• Lessons Learned
• It intends to DETECT if certain risks are presented during the maneuver execution.
Because of that some people might consider it to be vague.
• It intends to be as simple as possible and was designed ONLY for test point risk
evaluation.
EMBRAER – Safety Review Board“Three Flags” Application to Test Point Execution
THREE FLAGS
Test Point Execution SCALE
I – Detect training or gradual approach necessities
Training
Try-outs
Gradual Approach
Can the lack or
necessity of these
affect Safety ?
Is Needed ? And
II – Detect Errors Tolerances
Test Tolerances
Positioning TolerancesDo
Affect safety if extrapolated or disregarded ?
AndCan be Considered to be tight ?
III – Detect Recovering or Discontinuing possibilities
Recovering
DiscontinuingWhen The maneuver, is there a probable chance to get
into unsafe situation ?
EMBRAER – Safety Review Board
System Failures
EMBRAER – Safety Review Board
System Failures Analysis
• Based on AC 1309 safety assessment used by Embraer.
• Takes credit of the System Safety Assessment reports for all • Takes credit of the System Safety Assessment reports for all maneuvers analyzed.
• Takes credit of the System Safety Assessment reports to obtain failure probabilities.
• Flight test crew must define failures effects.
EMBRAER – Safety Review Board
FIN
AL P
RO
BA
BIL
ITY
HIGHLY PROBABLE
P > 10e-3LOW MEDIUM HIGH UNACCEPTABLE UNACCEPTABLE
PROBABLE
10e-3 > P > 10e-5LOW LOW MEDIUM HIGH UNACCEPTABLE
REMOTE
10e-5 > P > 10e-7LOW LOW LOW MEDIUM HIGH
EXTREMALLY
REMOTE LOW LOW LOW LOW MEDIUM
TABLE – SYSTEM FAILURES EVALUATION
FIN
AL P
RO
BA
BIL
ITY
REMOTE
10e-7 > P > 10e-9
LOW LOW LOW LOW MEDIUM
IMPROBABLE
P < 10e-9LOW LOW LOW LOW LOW
NO SAFETY
EFFECTMINOR MAJOR HAZARDOUS CATASTROPHIC
FAILURE EFFECT
EMBRAER – Safety Review Board
Flight EnvelopeTest Point
ExecutionSystem Failures
ACJ 1309 evaluation
EMBRAER FLIGHT TEST RISK ASSESSMENT METHOD
Operational Envelope
Design
Three Flags Method
ACJ 1309 evaluation
combined with flight
test effects to each
failure
Sequence of Assessment
Limited
Screen Factor Method
Each analysis is independent from each other.
EMBRAER – Safety Review Board
Risk Management
EMBRAER – Safety Review Board
Risk Management
• VMCG
• HAZARD: Landing gear collapse.
• CAUSE(S): Shimmy and ground/flight loads;
• RISK MINIMIZATION:
-As many as you want.
-Emphasize main hazards of the test.
-To agree upon Minimizing and Mitigation procedures.
Ex.
• RISK MINIMIZATION:1. According to the theoretical predictions the new caster is conservative
in relation to shimmy. However, the new landing gear shimmy
characteristics were not simulated. A ground test with and without the
steering spring will be performed to access the shimmy and control
characteristics of the new nose landing gear.
2. The landing gear loads predicted are 1% higher than in previous
design.
• RISK MITIGATION1. Firefighters standing by.
2. Ambulance standing by.
and Mitigation procedures.
EMBRAER – Safety Review Board
GENERAL OBSERVATIONS – METEOROLOGICAL CONDITIONS
Risk ClassificationMETEOROLOGICAL CONDITION
(TEST AREA AND DESTINATION/ALTERNATIVE)
LOW VMC/IMC (according to test requirements)
MEDIUM VMC/IMC (according to test requirements)
HIGH VMC (necessary)
1ST FLIGHT VMC (necessary)
EMBRAER – Safety Review BoardGENERAL OBSERVATIONS – CREW MINIMUM REQUIREMENTS
ClassificationExperience
(years)
Total Flight
Experience
(flight hours)
Rest between
same type of
flight (hours)
Number of
Flights on the
same day
(same
classification)
Minimum
Pilot Crew
Parte 23
Minimum Pilot
Crew
Parte 25
LOW ≥ 1 ≥ 1.000 ≥ 2 ≤ 3 1 PPRA(1) 1 PPRA(1)
MEDIUM ≥ 2 ≥ 1.000 ≥ 4 ≤ 2 1 PPRA(1) 2 PPRA(1)
HIGH ≥ 5 ≥ 1.500 ≥ 12 ≤ 1 2 PPRA(1) 2 PPRA(1)
Pilots
HIGH ≥ 5 ≥ 1.500 ≥ 12 ≤ 1 2 PPRA 2 PPRA
1st Flight ≥ 10 ≥ 2.000 ≥ 12 ≤ 1 2 PPRA(1) 2 PPRA(1)
ClassificationExperience
(years)
Total Flight
Experience (flight
hours)
Rest between
same type of
flight (hours)
Number of
Flights on the
same day (same
classification)
FTE(3) Rate
LOW - ≥ 10 ≥ 2 ≤ 3 C
MEDIUM ≥ 1 ≥ 100 ≥ 4 ≤ 2 B
HIGH ≥ 3 ≥ 200 ≥ 12 ≤ 1 A
1stststst Flight ≥ 5 ≥ 500 ≥ 12 ≤ 1 A*
FTEs
EMBRAER – Safety Review Board
• The Risk Assessment method is been used for only 6 months so it is
VERY NEW.
CONCLUSIONS
• Search for inconsistencies during application of the method.
• Nothing can substitute the FTE/TP discussions and analysis before the
actual flight test.
• The Risk Assessment method intends to be a trustworthy guide for
flight test preparation.
EMBRAER – Safety Review Board
EMBRAER – Safety Review Board
Embraer Risk Assessment
“Three Flags” Method
Maximilian Kleinubing – B.S.
Flight Test Engineer
Embraer Gavião Peixoto
email: [email protected]
THANK YOU !
EMBRAER – Safety Review Board
Example
High Speed
Characteristics
Risk Assessment
(abstract)
Version: - DEN/GFT – 22 / 07 / 2009
EMBRAER – Safety Review BoardRisk Assessment: High Speed Characteristics
EMBRAER – Safety Review Board
Risk Assessment
• The Risk Assessment here presented was based on the Embraer
Normative System ENS-000650 Rev. 5 (Gerenciamento de Riscos em
Ensaio em Vôo).
The risk will be evaluated in the following items:
• System Failure
• Flight Envelope
• Test Point Execution
EMBRAER – Safety Review Board
• Risk Classification:
MANEUVER SF TPE FE RATEFINAL RATE FINAL RATE FINAL RATE FINAL RATE
AFTER AFTER AFTER AFTER MITIGATIONMITIGATIONMITIGATIONMITIGATION
Vdf-Mdf Determination L M M M M
CG Shift L L M M M
Inadvertent Speed Increase L M M M M
Lateral Gust Upset L L M M M
Risk Assessment: High Speed Characteristics
Lateral Gust Upset L L M M M
Longitudinal Gust Upset L L M M M
Two Axis Gust Upset L M M M M
Leveling Off From Climb L L M M M
Descent From Mach Airspeed Limit Altitude L L M M M
Speed Brake Deployment (up to Vfc / Mfc) L M M M M
Speed Brake Deployment (Vdf / Mdf) L H H H H
Roll Capability (up to Vfc / Mfc) L M M M M
Roll Capability (Vdf / Mdf) L H H H H
Speed Excursion to VDF/MDF L M M M M
EMBRAER – Safety Review Board
High Speed Characteristics
Hazard: Speed excursion beyond Vd/Md
Cause: Excessive pitch down associated to loss of longitudinal control effectiveness
Minimizing risk procedures:1. Flutter Flight Test Campaign already performed up to Vd/Md.
Emergency procedures:Emergency procedures:1. Retard both engines thrust levers to IDLE;
2. Speed brake (If Speed Brake Deployment test already executed);
3. Apply primary longitudinal control (elevator) nose up;
4. Apply secondary longitudinal control (pitch trim) nose up;
5. Return for landing.
EMBRAER – Safety Review Board
High Speed Characteristics
Hazard: Speed excursion beyond Vd/Md
Cause: Loss or Misleading Airspeed Information
Minimizing risk procedures:1. Crew must monitor and compare different airspeed sources;
Emergency procedures:Emergency procedures:1. Retard both engines thrust levers to IDLE;
2. Apply primary longitudinal control (elevator) nose up;
3. Apply secondary longitudinal control (pitch trim) nose up;
4. Return for landing.
EMBRAER – Safety Review Board
Conclusions:
MANEUVER SF TPE FE RATEFINAL RATE FINAL RATE FINAL RATE FINAL RATE
AFTER AFTER AFTER AFTER MITIGATIONMITIGATIONMITIGATIONMITIGATION
Vdf-Mdf Determination L M M M M
CG Shift L L M M M
Inadvertent Speed Increase L M M M M
Lateral Gust Upset L L M M M
Risk Assessment: High Speed Characteristics
Lateral Gust Upset L L M M M
Longitudinal Gust Upset L L M M M
Two Axis Gust Upset L L M M M
Leveling Off From Climb L L M M M
Descent From Mach Airspeed Limit Altitude L L M M M
Speed Brake Deployment (up to Vfc / Mfc) L M M M M
Speed Brake Deployment (Vdf / Mdf) L H H H H
Roll Capability (up to Vfc / Mfc) L M M M M
Roll Capability (Vdf / Mdf) L H H H H
Speed Excursion to VDF/MDF L M M M M
EMBRAER – Safety Review Board
Conclusions:
1. According to the Risk Assessment here presented, the risk involved in the Roll
Capability and Speed Brake Deployment at Vdf / Mdf was considered HIGH
2. The pilot and FTE must:
CR PILOT/FTEExperience
(years in the
Flight Test Experience
Rest Time between
Number of Flight in the
PILOT / FTE
Risk Assessment: High Speed Characteristics
3. VFR (Daylight) – Finish High Risk Test Points 1h before sunset.
4. The Roll Capability and Speed Brake Deployment at Vdf / Mdf requires: Ambulance (Medical Support) and SAR.
CR PILOT/FTE (years in the
activity)Experience
(hours)
between Flights
Flight in the same day
(ENS-004757)
HIGHPILOT ≥ 5 ≥ 1.500 ≥ 12 ≤ 1 2 PPRA
FTE ≥ 3 ≥ 200 ≥ 12 ≤ 1 A
EMBRAER – Safety Review Board
Conclusions (cont.):
1. According to the Risk Assessment here presented, the risk involved in the High Speed
Characteristics Campaign (except Roll Capability and Speed Brake Deployment at Vdf
/ Mdf ) was considered MEDIUM
2. The pilot and FTE must:
Experience Flight Test Rest Time Number of PILOT / FTE
Risk Assessment: High Speed Characteristics
3. VFR or IFR, according to test requirement and procedure during recovery of the
maneuver.
4. The Campaign does not require special support (Firefighters, Chase and SAR).
CR PILOT/FTEExperience
(years in the
activity)
Flight Test Experience
(hours)
Rest Time between Flights
Number of Flight in the same day
PILOT / FTE (ENS-004757)
MEDIUMPILOT ≥ 2 ≥ 1.000 ≥ 4 ≤ 2 1 PPRA
FTE ≥ 1 ≥ 100 ≥ 4 ≤ 2 B
EMBRAER – Safety Review Board
System Failure Risk Classification:
MANEUVERMANEUVERMANEUVERMANEUVER BASIC EVENTBASIC EVENTBASIC EVENTBASIC EVENT FAILUREFAILUREFAILUREFAILURERATERATERATERATE
FAILUREFAILUREFAILUREFAILUREEFFECTEFFECTEFFECTEFFECT
RISKRISKRISKRISKASSESSMENTASSESSMENTASSESSMENTASSESSMENT
Total loss of airspeed information 1.00E-07 HAZARDOUS LOW
Misleading display of airspeed information on
both primary flight displays1.00E-07 HAZARDOUS LOW
Loss of airspeed information on primary flight
displays1.00E-05 MAJOR LOW
All Maneuvers
displays
Misleading display of airspeed information on
one primary flight display1.00E-05 MAJOR LOW
Pitch trim runaway beyond timer limits 5.11E-12 CATASTROPHIC LOW
Pitch trim runaway within timer authority 3.90E-07 MAJOR LOW
Total Loss of pitch trim Function N/A MINOR LOW
Asymmetric Uncommanded spoiler panels
deployment beyond monitor limits 6.83E-10 CATASTROPHIC LOW
Symmetric Uncommanded spoiler panels
deployment beyond monitor limits 3.42E-10 MAJOR LOW
Asymmetric spoiler panel upfloat N/A MAJOR LOW
EMBRAER – Safety Review Board
Test Point Execution Risk Classification - Three Flags:
MANEUVERMANEUVERMANEUVERMANEUVERQUESTIONSQUESTIONSQUESTIONSQUESTIONS
FLAGSFLAGSFLAGSFLAGS RISK RISK RISK RISK ASSESSMENTASSESSMENTASSESSMENTASSESSMENT1111 2222 3333
Vdf-Mdf Determination - 2 MEDIUM
CG Shift - - - 0 LOW
Inadvertent Speed Increase - 2 MEDIUM
Lateral Gust Upset - - 1 LOWLateral Gust Upset - - 1 LOW
Longitudinal Gust Upset - - 1 LOW
Two Axis Gust Upset - - 1 LOW
Leveling Off From Climb - - - 0 LOW
Descent From Mach Airspeed Limit Altitude - - 1 LOW
Speed Brake Deployment (up to Vfc / Mfc) - 2 MEDIUM
Speed Brake Deployment (Vdf / Mdf) 3 HIGH
Roll Capability (up to Vfc / Mfc) - 2 MEDIUM
Roll Capability (Vdf / Mdf) 3 HIGH
Speed Excursion to VDF/MDF - 2 MEDIUM
EMBRAER – Safety Review Board
Flight Envelope Risk Classification: Screen Factor
MANEUVERMANEUVERMANEUVERMANEUVERQUESTIONSQUESTIONSQUESTIONSQUESTIONS
FLAGSFLAGSFLAGSFLAGS RISK RISK RISK RISK ASSESSMENTASSESSMENTASSESSMENTASSESSMENT1111 2222 3333
Vdf-Mdf Determination - - 1 NOT HIGH
CG Shift - - - 0 NOT HIGH
Inadvertent Speed Increase - - 1 NOT HIGH
Lateral Gust Upset - - 1 NOT HIGHLateral Gust Upset - - 1 NOT HIGH
Longitudinal Gust Upset - - 1 NOT HIGH
Two Axis Gust Upset - - 1 NOT HIGH
Leveling Off From Climb - - - 0 NOT HIGH
Descent From Mach Airspeed Limit Altitude - - 1 NOT HIGH
Speed Brake Deployment (up to Vfc / Mfc) - 2 NOT HIGH
Speed Brake Deployment (Vdf / Mdf) 3 HIGH
Roll Capability (up to Vfc / Mfc) - - 1 NOT HIGH
Roll Capability (Vdf / Mdf) 2 HIGH
Speed Excursion to VDF/MDF - - 1 NOT HIGH
EMBRAER – Safety Review Board
Flight Envelope Risk Classification:
TEST POINTTEST POINTTEST POINTTEST POINT FLIGHT FLIGHT FLIGHT FLIGHT ENVELOPEENVELOPEENVELOPEENVELOPE
RISK RISK RISK RISK CLASSIFICATIONCLASSIFICATIONCLASSIFICATIONCLASSIFICATION
Vdf-Mdf Determination DESIGN MEDIUM
CG Shift DESIGN MEDIUM
Inadvertent Speed Increase DESIGN MEDIUM
Lateral Gust Upset DESIGN MEDIUMLateral Gust Upset DESIGN MEDIUM
Longitudinal Gust Upset DESIGN MEDIUM
Two Axis Gust Upset DESIGN MEDIUM
Leveling Off From Climb DESIGN MEDIUM
Descent From Mach Airspeed Limit Altitude DESIGN MEDIUM
Speed Brake Deployment (up to Vfc / Mfc) DESIGN MEDIUM
Speed Brake Deployment (Vdf / Mdf) 3 FLAGS HIGH
Roll Capability (up to Vfc / Mfc) DESIGN MEDIUM
Roll Capability (Vdf / Mdf) 3 FLAGS HIGH
Speed Excursion to VDF/MDF DESIGN MEDIUM
EMBRAER – Safety Review Board
System Failure:
FIN
AL
PR
OB
AB
ILIT
Y
HIGHLY
PROBABLELOW MEDIUM HIGH UNACCEPTABLE UNACCEPTABLE
PROBABLE LOW LOW MEDIUM HIGH UNACCEPTABLE
REMOTE LOW LOW LOW MEDIUM HIGH
EXTREMALLY
REMOTELOW LOW LOW LOW MEDIUM
IMPROBABLE LOW LOW LOW LOW LOW
NO SAFETY
EFFECTMINOR MAJOR HAZARDOUS CATASTROPHIC
FAILURE EFFECT
EMBRAER – Safety Review Board
Embraer Risk Assessment
“Three Flags” Method
Maximilian Kleinubing – B.S.
Flight Test Engineer
Embraer Gavião Peixoto
email: [email protected]
THANK YOU !
EMBRAER – Safety Review BoardExample:
Test Point – Maximum Dive Speed Demonstrated in Flight (Vdf) Determination
Aircraft: Choose an aircraft you are currently flying.
Configuration: UP/0
Initial Altitude: 15.000 ±1000 ft
Trim Speed: Vmo
“Three Flags” Application to Test Point Execution“Three Flags” Application to Test Point Execution
Trim Speed: Vmo
Minimum Recovery Altitude: 10.000 ft
1. TRIM aircraft at Vmo/Mmo during 10 seconds;
2. ESTABLISH a pitch attitude 7.5° (pushover maneuver Nz=0.5g) below the trimmed
attitude and maintain during 20 seconds. Do not reduce thrust;
3. PERFORM a pull up maneuver with load factor 1.5g. After start pull up, THRUST
REDUCTION may be used, if required.