Human and Automation/Robotic

Integration in Spaceflight: Design &

Operational Challenges Jessica J. Marquez, Ph.D.

Jessica.J.Marquez@nasa.gov NASA Ames Research Center

Human-Systems Integration Division

1

Human & Automation/Robotic Integration Goal: •  Provide overview of Human and Automation/Robotic Integration (HARI) as a risk •  Present the evidence for this risk from the context of the Human Research

Program •  Highlight HARI research direction

2

Space Human Factors Engineering: Risk of Inadequate Design of HARI !  Risk statement: Given that automation and robotics must seamlessly

integrate with crew, and given the greater dependence on automation and robotics in the context of long-duration spaceflight operations, there is a risk that systems will be inadequately designed, resulting in flight and ground crew errors and inefficiencies, failed mission and program objectives, and an increase in crew injuries.

Human Research Program

Space Human Factors & Habitability

Space Human Factors Engineering

HAB TRAIN HCI TASK HARI 3

What does the future hold? !  NASA Design Reference

Missions (DRMs) present a myriad of future human exploration missions.

!  Game-changers: !  Fewer crewmembers !  Farther away destinations !  Longer duration missions !  Variant, intermittent

communication delays !  Crew autonomy !  Less ground support

futu

re m

issio

ns

More automation & robotics 4

Exploration Missions !  Game-changers will shift the way we do human

spaceflight operations.

!  NASA will have to build upon & go beyond its existing human spaceflight operational experience, which has heavily relied on ground control support.

!  NASA will have to infuse existing automation/robotic technology, which need to be validated in safety-critical context.

!  Future human spaceflight will be more than developing automation/robotic technology – it will have to be about integrating these technologies with people.

5

Benefits and Consequences of Automation & Robotics

Consequences Benefits

Lower workload

Increased efficiency

Increased capabilities

Unexpected vulnerabilities

Changing nature of work

6

How do we know that human and automation/robotics integration is challenging?

Credit: MIT Instrumentation Laboratory Report (circa 1960s)

7

Wondering since 1950s …Fitt’s List

Attribute Machine Human

Speed Superior Comparatively slow Power output Superior in level in consistency Comparatively weak Consistency Ideal for consistent, repetitive

action Unreliable, learning & fatigue a factor

Information Capacity

Multi-channel Primarily single channel

Memory Ideal for literal reproduction, access restricted and formal

Better for principles & strategies, access versatile & innovative

Reasoning Computation

Deductive, tedious to program, fast & accurate, poor error correction

Inductive, easier to program, slow, accurate, good error correction

Sensing Good at quantitative assessment, poor at pattern recognition

Wide ranges, multi-function, judgment

Perceiving Copes with variation poorly, susceptible to noise

Copes with variation better, susceptible to noise

Hollnagel, 2000

8

Inadequate HARI Design

! We can see other examples of when automation and robotics were introduced within space and aeronautics domain.

9

Aviation Accidents and Incidents !  FAA report (1996) identified vulnerabilities in flightcrew

management of automation and situation awareness.

!  Flightcrew/flight deck automation interactions contributing to accidents !  Airbus A300-600 (China Airlines, Japan): lack of mode

awareness resulting in conflicting actions taken by flight crew and autopilot. (Sogame & Ladkin, 1996)

!  Boeing 757 (American Airlines, Columbia): transitioning from high to low levels of automation, coupled with incorrect inputs into flight management system, resulted in loss of situation awareness. (Endsley & Strauch, 1997)

!  Recently, inadequate automation interactions have been associated with accidents & incidents: !  Boeing 777-200ER (San Francisco, CA): contributing were pilot

interactions with complexity of autothrottle & autopilot flight director, inadequate pilot mental model. (NTSB Aircraft Accident Report, 2014)

10

Human Factors Analysis & Classification System

Design of automation & robotics technology

Shappell & Wiegmann (2000)

Improper choices, misinterpretation and/or misuse of relevant information

Human Factors Analysis & Classification System (HFACS) has been used to investigate and classify human error in a variety of high-risk settings.

11

HFACS Contributors in Aviation Accidents !  Commercial aviation accident data from 1990 – 2002 (NTSB &

FAA databases): 1020 accidents (Shappell et al., 2007) !  Accident descriptions were identified as having one or more of the

HFACS contributors

!  Preconditions for Unsafe Acts, associated with Technological environment: 15 accidents (1.5%)

!  Unsafe Acts of the Operator associated with Decision errors: 374 accidents (36.7%)

!  Similar study of Republic of China Air Force accidents from 1978 – 2008: 545 accidents (Ting & Dai, 2011) !  Technological environment: 48 accidents (9.0%) !  Decision errors: 233 accidents (43.9%)

!  These results do not mean that HARI is a main contributor to aviation accidents, but rather associated human factors issues are related to HARI.

12

Evidence from Research What We Imagine Reality Check

!  Using Automation may lead to:

!  Inability to maintain mode awareness

!  Decreased situation awareness

!  Mode-related errors

!  Skill degradation

!  Inappropriate knowledge acquisition

!  Lack of trust (disuse of automation)

!  Complacency and system overreliance

!  Errors of omission and commission

!  Decision/automation bias Credit: Marvel Studios, Iron Man & The Avengers

13

“Right” amount of automation? High 10 The computer decides everything, acts autonomously,

ignoring the human.

Leve

ls o

f Au

tom

atio

n (

LOA

)

9 Informs the human only if it, the computer, decides to

8 Informs the human only if asked, or

7 Executes automatically, then necessarily informs the human, and

6 Allows the human restricted time to veto before automatic execution, or

5 Executes the suggestion if the human approves, or

4 Suggests one alternative

3 Narrows the selection down to a few, or

2 The computer offers a complete set of decision/action alternatives, or

Low 1 The computer offers no assistance, human must take all decisions and actions

Completely autonomous

Management by exception

Collaborative automation

Completely manual control

Based on Sheridan & Verplank (1978), Wickens et al., 2000

14

No magic bullet solution !  LOA provide a framework to discuss, not actually

guidance. The type of activity and functions are important in design. (Bradshaw et al., 2013)

!  Balancing Act: increase needs for capabilities that automation and robotics affords while mitigating consequences. !  Better recovery from automation failures when the

level of automation during the task involved human interaction. (Endsley & Kiris, 1995)

!  Increasing amount of automation supports routine system performance and workload, but negatively affects failure system performance and situation awareness. (Onnasch et al., 2013)

15

HARI in Spaceflight !  Human Factors Engineering plays an important role

in design and operations of spaceflight.

!  With vehicle systems becoming more complex, interface and automation systems are intertwined, making HARI design a key part of vehicle design. !  Recent reminder: NTSB is adding human performance

group to look at “the interface between flight crew and the vehicle” for SpaceShipTwo accident (2014).

!  Key Point: Having operational experience on a particular HAR system does not necessarily mean that the system can be seamlessly transported to a different mission nor that it is robust to that mission.

16

Apollo 10: Mode Awareness !  Testing separation of lunar lander’s two stages over the

lunar surface. !  After separation, crew has to rendezvous back to command

module.

!  One crew switched mode of guidance and navigation system. !  Unaware of mode change, second crew switched again. !  Result: firing thrusters, gimbal lock, tumbling. !  Crew response: switching back to correct mode, override

computer and took over manual control to successfully recover.

!  Crew made 8 complete rolls in 15 seconds.

!  Post-hoc analysis estimates that they were 2 seconds away from crashing on lunar surface (Shayler, 2000)

https://www.youtube.com/watch?v=g5N2pygq42A

17

Operational Experience: Spacecraft Docking !  Since Apollo through ISS, spacecraft rendezvous and docking

have been an essential component to human spaceflight operations.

!  Operations has been mostly automated but has included human-in-the-loop as a backup and/or as a critical task executor.

!  Over time, docking has become more complex and the methods of attaching spacecraft to each other has evolved.

18

Spacecraft Docking: Mir !  Crew testing manual backup docking system.

!  Near Collision 1997: “a near collision with a resupply cargo ship during a manual docking system test.”

!  Accident 1997: Progress collided with a Mir module, depressurized it, and severely damaged solar arrays.

!  “What [operator] was seeing on his screen was an image that didn’t change in size very fast… He couldn’t determine accurately from the image that the speed was too high.” (Morgan 2001)

Morgan, C. (2001) “Shuttle-MIR: The United States and Russia Share History’s Highest Stage” NASA-SP-4225, NASA Johnson Space Center, Houston, TX.

19

Operational Experience: Robotic Arms

Timelapse Video of Cygnus Release: http://youtu.be/-dtOS-oavGg

EVA

Dextre

Visiting Vehicles

ISS Robotics Workstation (RWS)

20

Crew and Ground Space Robotics !  In practice, two crewmembers use Robotic

Workstation: primary controller while second crew is dedicated to managing camera operations, observing procedures, and confirming direction of motion. !  Increases their situation awareness

!  Maintain a system of checks and balances

!  Comments for improved camera views and assistive overlays.

!  In order to off-load crew time, ground controllers handle a lot of the pre-positioning of robotic tasks.

Flight Crew Integration (FCI) ISS Life Sciences Crew Comments Database

21

Dextre: Special Purpose Dexterous Manipulator (SPDM) !  Choreographed from ground.

!  Designed & implemented knowing that timelines would be excessive and beyond available crew resources.

!  Uses automated sequences commands.

!  There is no direct operator viewing & requires large number of procedural steps.

!  Has limited ability to respond to real-time anomalies, requiring day/s to re-plan.

!  Two, seven-jointed robotic arms !  Arrived on ISS in 2008, EVA

crewmembers assembled. !  First operational task: 2011. 22

Commanding Space Robotics Agents: rovers & spaceships !  Rovers/Landers on Mars

!  “Operations are open-loop, where the human must send sequences of commands rather than act on fed-back information in real-time due to the long signal time delays between Earth and Mars”

!  Commands to ISS !  Space Station is monitored

& commanded by a team of flight controllers, each with their specialization.

!  Everything from power management to attitude control.

Mars Science Lab Scientists & Engineers Planning A Day

NASA Mars Mission ISS Mission Control Center, Front Room

23

Building Upon Experience Open questions about robotic arms and rover operational experience:

Do we have a human-robotic system that supports efficient and effective task execution?

Do we have a system that an astronaut can use on Mars or do we have a system that ground can use?

24

Inadequate HARI Design !  What are the root causes for

inadequate HARI design?

!  From the Human Research Program perspective, we investigate three research areas to mitigate the risk to crew for current and future spaceflight missions.

!  “New technology does not remove human error. It changes it.” (Dekker, 2006)

!  Automation is only as good as we build it. !  It inherently is imperfect

and incomplete, because our knowledge of complex, new system behavior & extraterrestrial environments is incomplete.

!  Humans are often considered the primary backup.

25

SHFE HARI Research Gaps !  SHFE-HARI-01: We need to evaluate, develop,

and validate methods and guidelines for identifying human-automation/robot task information needs, function allocation, and team composition for future long duration, long distance space missions.

Information Needs & Allocations

System & Interaction Design

Human-System Performance

26

SHFE HARI Research Gaps !  SHFE-HARI-02: We need to develop design

guidelines for effective human-automation-robotic systems in operational environments that may include distributed, non-colocated adaptive mixed-agent teams with variable transmission latencies.

Information Needs & Allocations

System & Interaction Design

Human-System Performance

27

SHFE HARI Research Gaps !  SHFE-HARI-03: We do not know how to quantify

overall human-automation-robotic system performance to inform and evaluate system designs to ensure safe and efficient space mission operations.

Information Needs & Allocations

System & Interaction Design

Human-System Performance

28

HARI Challenges for Exploration Missions !  Long & intermittent communication time-delays

!  Tele-operations and autonomous commanding of robotic agents at variant distances

!  Supervisory control of complex, automated vehicle systems

!  Variety of mixed-agents, different types of automation & robotic agents

!  Human-robot team coordination

!  Crew autonomy, planning and execution

29

HARI Challenges for Exploration Missions !  Long & intermittent communication time-delays

!  Tele-operations and autonomous commanding of robotic agents at variant distances

!  Supervisory control of complex, automated vehicle systems

!  Variety of mixed-agents, different types of automation & robotic agents

!  Human-robot team coordination

!  Crew autonomy, planning and execution

Key Points

Human & automation/robotic integration is important for the success of future

exploration missions.

HARI design needs to be robust, not necessarily perfect or optimized.

We need to transition HARI operational experience & lessons learned to make

them work for new missions. 30

Questions? http://humanresearchroadmap.nasa.gov/evidence/reports/HARI.pdf

31

Backups

32

!  Successful robotic operations will need operator to have knowledge of:

!  Overall awareness

!  Working conditions

!  Robot’s environment

!  State of robot

Future of Integrated Human-Robotic Interaction

!  HRI standards can be complex:

!  Robots developed with customized interfaces, customized methods of interaction

!  Level of coordination & control often highly task dependent

Keyes et al. (2010) Ferketic et al. (2006)

Standardizing types of operations & interactions necessitates common metrics and measures: fundamental commands, operations, & interfaces. 33