this resource is made publicly available to anyone seeking to adopt … · area, to illustrate the...

1

This resource is made publicly available to anyone seeking to adopt or design safer

and more usable wearable AR solutions. It is provided by the members of the

AREA at no cost to the ecosystem of Augmented Reality technology providers,

customers of AR-enabled solutions and others participating the advancement of

AR adoption.

1

© 2014 - 2018 All Rights Reserved

www.theAREA.org

AREA AR Safety Framework Case

Study

The AREA © 2018. All rights reserved.

This is a proprietary report prepared under contract with the AREA.

As part of the AREA’s support of the development of sound information and

best practices for the introduction and adoption of AR, access to and use of

this proprietary report is provided to all members of the enterprise AR

ecosystem. If you wish to make your partners, suppliers and customers

aware of this AREA research report, you may share this resource provided

that this information and the content of the report are not edited.

The entire content of this proprietary report is protected by copyright. No

part of this publication may be reproduced, stored in a retrieval system or

transmitted in any form or by any electronic, mechanical, photocopying and

recording means or otherwise, without the prior written permission of the

AREA.

2


www.theAREA.org


Study

TABLE OF CONTENTS

Purpose ...................................................................................................................... 3

Statement of the Challenge ........................................................................................ 4

Case Study of an Aerospace Assembly Use Case ................................................. 4

AR Project Cycle ........................................................................................................ 5

Step 1: Capture of Requirements ............................................................................... 6

Step 2: Design and Build ............................................................................................ 9

Step 3: Test .............................................................................................................. 12

Step 4: Deliver .......................................................................................................... 13

Looking Ahead ......................................................................................................... 14

3


www.theAREA.org


Study

PURPOSE

This document is prepared by The Manufacturing Technology Centre, on behalf of the

AREA, to illustrate the proposed steps of a solution provider when assessing their

solution for safety in a manufacturing assembly process.

This resource is made publicly available to anyone seeking to adopt or design safer

and more usable AR solutions. It is provided by the members of the AREA at no cost

to the ecosystem of Augmented Reality technology providers, customers of AR-

enabled solutions and others participating the advancement of AR adoption.

The steps in this case study make use of the AREA Safety and Human Factors

Assessment Framework Tool and the proposed risk assessment framework in the

AREA report. The AREA Safety and Human Factors Assessment Framework is a

working model made available exclusively to AREA members for assessing AR design

and hardware for safety and usability.

An accompanying AREA member exclusive report provides best practices for

wearable AR technology safety and human factors assessment in the enterprise.

This project and resultant model benefits project managers, solution providers,

developers and safety managers by:

- Encouraging collaboration between risk management and the project team, - Providing methods and tools for the assessment of safety and human factors,

that can be applied at the correct time, and aligned with the typical project lifecycle, and

- Offering specific approaches to mitigate risks that, in part due to lack of strong human factors backgrounds, may be introduced when AR is introduced in the workplace.

To learn more about joining the AREA, these reports and other member benefits,

please visit http://thearea.org and/or contact its Executive Director, Mark Sage.

http://thearea.org/

mailto:[email protected]

4


www.theAREA.org


Study

STATEMENT OF THE CHALLENGE

Manufacturing companies are increasingly seeking to automate complex manual

assembly tasks in order to improve quality, efficiency and reduce cost.

Due to the mix of machines and employees, manufacturing workplaces require

proactive enforcement of safety and risk management protocols. The introduction of

wearable AR solutions in a typical workshop environment requires another level of

careful assessment and identification of possible safety issues before adoption. Then

mitigation measures must be incorporated into the design stages of a safe solution.

Case Study of an Aerospace Assembly Use Case

Augmented Reality can be an effective tool to ensure that procedures are followed

consistently and accurately as part of quality management processes and in

compliance with industry regulations. In assembly use cases, AR delivers value by

presenting information digitally in the user’s context to guide them through complex

procedures. The costs and time associated with re-work and lack of compliance are

reduced when assembly is done correctly every time.

For this case study, an AR-assisted assembly project is commissioned by a company

that manufactures and assembles aerospace systems in compliance with strict

assembly regulations at each step. The work order instructions are accessed at a fixed

workstation, however, assembly can take place at another area of the factory floor or,

in the case of large assemblies, can require working at height. The company seeks to

improve operational efficiency by ‘augmenting the manual worker’ through the use of

portable AR-assisted devices.

An AR solutions provider is hired to build a solution designed to reduce the time taken

for complex assembly by delivering AR-assisted instructions. The solution must

prioritize safety and usability in its design to ensure the successful adoption by the

client and to be deployed on both a wearable AR display and a companion app running

on a tablet.

5


www.theAREA.org


Study

AR PROJECT CYCLE

A project cycle is used as a reference to describe recommended actions and

methods/tools to capture and mitigate for safety risks and usability. In this case study

and the AREA Safety and Human Factors Assessment Framework Tool, this cycle

has been referred to as the ‘AR project cycle’.

Shown in the figure below, the AR project cycle emphasizes recommended actions to

promote safety and usability within the proposed AR solution.

Figure 1. AR project cycle (source: AREA Safety and Human Factors Assessment Framework Tool).

The AR Project Cycle - Solutions Provider

Recommended Actions

Supporting Tools

Requirements Capture (Systems

Definition)Design Build Testing Training Delivery

• Identify SafetyRequirements by Defining:

• Context of use (process, task and environment)

• Currently used hardware,

software and equipment• Existing health and safety

hazards• Safety compliance and

regulatory factors• Key actors in the

project/implementation

• Define end-user demographics and characteristics

• Assess Solution Design for Safety and Usability

• Undertake risk assessment of solution design

• Identify mitigations for solution

• Assess and select device to safety requirements

• Define Safety and Usability Testing

• Define acceptable safety factors

• Conduct safety and Usability trials

• Analyse results andincorporate feedback into design

• Document Solution Specification for the End-User Including:

• Instructions on proper use of solution

• Known risks• Residual risks

• AR Device Assessment

• AR Design Assessment

• AR Generic Risks Table

(Ref)

• AR Design Assessment

• AR Generic Risks Table (Ref)

• Usability and Safety Tools (Ref)

6


www.theAREA.org


Study

STEP 1: CAPTURE OF REQUIREMENTS

During this stage, the project team or manager describes the technology, the

processes to be modified and other factors directly or indirectly relevant including:

● The task, the process and the environment, ● Hardware, software and equipment currently in use, ● Health and safety hazards

o Through discussions with the client’s side safety/human factors manager and/or project manager,

o Through current risk management documents on the assembly process such as risk assessments or process failure mode effects analysis (PFMEAs)

● Safety compliance and regulatory factors including: o Known country/state regulations, o Organizational policies,

● Key actors in the project/implementation, ● End user demographics.

Table 1. Example of requirements capture process output.

Project Requirements Summary

Task/process Assembly of engine components. Manufacturer has defined the six work orders

in the process for assembling components on an aircraft engine. Each work

order is then broken down into smaller tasks known as “operations”

Environment Typical workshop environment.

AR location of use Working at height for some tasks requiring climbing of scaffolding.

Mainly working on various areas of shop floor.

Hardware, software and

equipment currently used

Work instructions are accessed through a fixed work station computer. Some

tools required specific assembly operation.

Existing health and safety

hazards

Risk assessments and PFMEA are available.

Safety/Compliance

• Personal Protective Equipment o OSHA 1910.132

• Ergonomics o OSHA CFR 1910 General Duty Clause, Section 5(a)(1)

• Fall Protection (ensuring protection from fall hazards) o OSHA 1910 Subpart D State and organizational policies

benchmarking

Key actors in

project/implementation

Solutions provider: project manager, development team Client –side: project manager, safety manager, human factors engineer.

User demographics (i.e. to

create end user personas,

and understand known

behaviors of end users)

Range between young apprentices to long-service employees, some workers

wear prescription lenses.

7


www.theAREA.org


Study

AR Assessment Tools

The ‘AREA Safety and Human Factors Assessment Framework’ produces results of

two assessments: one focusing on the AR device and another on design of the

solution. These tools, consisting of a set of comprehensive questions and related

information, support the key roles involved in a project to systematically identify and

derive safety and human factors related issues and provide high level

recommendations to include in the design of the solution. The tools assist in analyzing

all the potential safety and usability requirements that may need consideration

surrounding the user, the environment, the context (the task) and its interactions within

the system.

AR Device Assessment Tool

This tool assists in defining device requirements for the solution. For this assembly

use case, the following are some constraints derived from the assessment:

Environment

● The workshop floor is brightly lit. ● The workshop floor can be noisy. ● Operators must be aware of hazards in their surroundings such as forklifts. ● Operators must be aware of guarding and zone exclusion zones whilst using

the device. ● Health and safety notes should be displayed to the device before operation as

current standard process.

Use case

● Assembly operators are required to work at height for some operations (e.g., climbing scaffolding and ladders).

● Hanging cables from the device may get caught/detached in the assembly process.

● Some assembly operations require the use of small power tools (e.g., automated nut runners and hand-drills).

User

● Some operators wear prescription glasses. ● Experienced users will need to use the device for complex assembly operations

(such as when assembling harnesses). Less experienced operators may require longer use of the device even for simpler operations as they’re training.

● Device needs to be comfortable to wear for duration required- each operation lasts approx. 20mins.

● Voice interaction with the device would be ideal leaving hands-free.

System

● Device needs to receive safety prompts and alarms.

8


www.theAREA.org


Study

A sample of the recommendations generated by the framework is in Table 2.

Table 2. Sample of device assessment results (source: AREA Safety and Human Factors Assessment Framework Tool).

Notes Device Safety Considerations

Main AR

Risk/Hazard

Will the user be

working at

height?

Working at height can be a

high-risk situation and may

require full attention, and

situational awareness. A

device obscuring user's FOV

should be avoided.

Consider how the device's form factor

and fit for user may increase

discomfort/distraction/obstruction to

user view. Can the device be used to

alert the user when they are in a

dangerous orientation/location?

Low Peripheral

Vision

Does the device

need to

especially

robust/ruggedize

d?

There are risks of damaging

the device whilst not worn by

the wearer. Device can be

dropped or handled rough

that could cause damage to

the external materials or

possibly internal circuitry.

Consider how the device will be used

by the user and what its resting place

will be normally. Will the device be

carried or moved around?

These may indicate that a ruggedized

device would be required to prevent

damage over time.

Asset damage

AR Design Assessment

Similarly, the design assessment component of the framework supports a developer

in designing a safe AR-assisted solution. For this assembly use case, the assessment

advises the Solutions Provider to consider the following when designing the AR

solution:

Environment

● The workshop is noisy - the design needs to consider multimodal interaction (i.e. not relying only on voice). The solution may also need to filter out noise so that safety critical alerts such as alarms can be heard.

● There is risk of asset and equipment damage in the surrounding environment. There are other components surrounding the operator such as equipment and materials stacked on shelves.

● The operator will be required to use the solution daily for intervals at least 20 minutes in duration. Design must consider reducing risk of eye strain.

User

● The users have some visual impairments (e.g., an operator is color blind). Designer will need to ensure that the content will be visible, despite visual impairments.

● Operators need to collaborate/communicate with other personnel, equipment or machines. Some operations in assembly require working in pairs. Shared experiences may be required.

● The solution must be intuitive and easy to set-up/calibrate. ● The solution must not obstruct the user’s field of view or peripheral view. ● The solution must not distract the user from hazards/risks posed to them in the

environment.

9


www.theAREA.org


Study

Use case

● Some operations will require AR-assisted work instructions whilst operating power tools. A solution must have a hands-free way of interacting with the AR device.

System

● The solution must allow safety critical information (e.g., alarms, alerts or prompts) to be received by the user.

● The solution must provide the user continual device status (e.g. battery status, offline/online status or malfunction).

A sample of the recommendations are found in Table 3 below:

Table 3. Sample of design assessment results (source: AREA Safety and Human Factors Assessment Framework Tool).

Notes Device Safety Considerations

Main AR

Risk/Hazard

2.3 Do the end users

have any experience

with using AR?

Introduction to the emerging

technology and training must be

considered before or as part of the

implementation phase. This should

include correct fitting/mounting of

device and use of functionality.

Reluctance to adapting new

technology may be an issue.

Consider appropriate calibration set

up.

Guided experience may be

required for new users or users

with little experience with new

technology interfaces. Design flow

may need to be intuitive and user

experience comfortable. Consider

a change management plan.

Inadequate training

3.2 Is the device

required to receive

safety critical

messages?

Alarms, messages require to be sent

and viewed by the headset by third

party IoT devices, then this

information/data needs to be

presented in the most effective way

to the user.

Consider how this

information/data will be displayed

effectively and safely. Urgent

safety information visual/tactile

may need to capture user's

attention without obstructing their

FOV while being in the midst of a

task.

Safety critical

communication

3.3 Does the user

have/require visibility

of the system? E.g. if

system is offline

If device loses signal or

malfunctioning, the user being

unaware of this could be a risk

Consider how the user will have

visibility of the current device

status - battery, malfunctioning,

offline status etc.

Safety critical

communication

STEP 2: DESIGN AND BUILD

There are no specific design standards for creating AR solutions, however, companies

adopt their own systems design methodology. Typically, the design cycle is iterative

and consists of a concept stage and detailed design stage.

In the concept stage of the project, the development team needs to ensure the design

meets the safety/usability requirements defined and prioritized in the first stage. To

10


www.theAREA.org


Study

ensure the design is safe and robust for the application, standard practice dictates that

risk assessments are undertaken before the final build/implementation of the design.

A list of commonly used and standardized risk assessment methods are provided in

the AREA Safety and Human Factors Assessment Framework Tool. A sample of the

tools captured are shown below in Table 4.

Table 4. Sample of risk assessment tools (source: AREA Safety and Human Factors Assessment Framework Tool)

Assessment

Domain Tool Metrics When to use them in Design Cycle?

Failure Modes

and Effects

Analysis

Breakdown of

failures, scored by

severity and

likelihood

Design - Concept Usually conducted once during the technology

cycle stages

HAZOP

System is divided into

subparts/subsystems

and analyzed one at

a time. Severity and

Likelihood scored.

Risk is then ranked.

Implementation -

Early

Systematic search for hazards which are defined

as deviations within these parameters that have

dangerous consequences.

Risk

Assessment

Severity and

Likelihood of risk is

scored

Implementation-

Early

Document systematically, outlining the known

hazards, risks and controls for AR

The following (Table 5) is an example output of an FMEA for the current assembly use

case.

11


www.theAREA.org


Study

Table 5 FMEA example (source: MTC, 2018)

Process

Step

Potential

Failure

Mode

Potential

Failure

Effect

SEV Potential

Causes

Current

Process

Controls

OCC DET RPN1 Action

Recommended

What is

this step?

In what

ways can

the step

go wrong?

What is

the impact

on the

user?

How

severe

is the

effect?

What

causes the

step to go

wrong?

What are

the

existing

control

that

prevent/de

tect the

failure

mode from

occurring?

How

frequent

ly is the

cause

likely to

occur?

How

prob

able

is

dete

ction

of

failur

e

mod

e?

SEV x

OCC x

DET

What are the

actions for

reducing the

occurrence of

the cause or

improving the

detection?

Operator

wears

headset

Headset

is not

calibrate

d

Experien

ce is

incorrect

7 User is not

aware of

calibration

process

n/a 6 5 210 Train user in

setting up

headset or

prompt user in

application to

calibrate

headset on

first use.

Operator

climbs

scaffoldin

g to

install

compone

nt

AR

content

obstruct

user view

Operator

visibility

is

reduced,

risk of

tripping

8 User does

not switch

off AR

whilst

climbing

scaffolding.

User keeps

headset on

while

climbing

scaffolding

n/a 6 7 336 AR content to

automatically

switch off in

task mode.

User prompted

to take off

headset before

climbing.

Operator

goes to

store to

get a tool

FOV is

obstructe

d, SA is

reduced

Distractio

n from

hazards

in

environm

ent –

moving

vehicles,

trip

hazards

7 User keeps

headset

and AR

experience

active

n/a 7 7 343 AR content to

automatically

disappear

when away

from work

zone. AR

headset

tracked and

alarm/prompts

when outside

work zone.

When doing risk assessment and defining acceptable levels of risk during the solution

design stage, a multi-disciplinary team is recommended. Team members may include:

● A person highly familiar with the assembly steps (such as the production manager or end user).

● A person (or same person as above) well-versed in the health and safety risks

1 Risk priority number: The overall risk score of an event. It is calculated by multiplying the scores for severity,

occurrence and detection. An event with a high RPN demands immediate attention while events with lower RPNs

are less risky. Traffic lights color coding used to highlight severity of risk.

12


www.theAREA.org


Study

of the process (from client’s side such as the safety manager and/or human factors engineer).

● The AR solution’s development team. ● A professional familiar with FMEA expertise but no specific knowledge of the

project or processes. This process is iterative, and it will be repeated until the design adequately mitigates

for risks.

STEP 3: TEST

Testing is an essential step in the AR project cycle as this evaluates whether the AR

solution has met all its requirements, including those pertaining to the solution’s safety

and usability.

There are several methods to evaluate an AR solution including subjective and

objective measurements through human perception, observation and expert analysis.

A comprehensive evaluation includes a combination of methods and measures.

In this assembly use case, distraction, cognitive strain and potential restriction in field

of view are three overlapping usability concerns. Common usability tools that could be

adapted to evaluate AR solutions in the required domain include:

• Questionnaires such as the NASA-TLX to assess the perceived mental workload on the user.

• Observational methods such as Verbal Protocol that involves asking the end user to verbally explain what they see and what they do to understand the reasons behind their decisions whilst navigating through the AR. application/experience. This method also allows the observer to record the number of errors made by the user. For example, if the user closes a critical health and safety pop up in the AR application without reading it.

• Objective methods such as gaze tracking and physiological response measures could give an indication of whether the user’s focus is on the expected object/area/task.

Table 6 summarizes usability assessment methods that could be used in combination

to provide feedback on the potential of user distraction and mental workload.

13


www.theAREA.org


Study

Table 6. Usability Tools extracted from 'Usability & Safety Tools (Ref)' from Safety Framework Tool

Assessment

Domain Tool Metrics

When to

use them

in Design

Cycle?

Notes

Situational

Awareness

Eye-tracker

(Eye point of

gaze)

Heat map, areas of

interest, TTFF,

Fixation Sequences

Design -

Testing

To determine where the operator is

looking

Situational

Awareness

Think Aloud

(Verbal

protocol)

Error rate, user

behaviour,

observation

Design -

Testing

A method used to gain insight how the

user behaves whilst navigating through an

UX/UI/system

Mental

Workload

Assessment

NASA -TLX Subjective, 7 point

scale

Design -

Testing

Questionnaire used to assess mental

workload on user

A full list of safety and usability assessment tools can be found in the AREA Safety

and Human Factors Assessment Framework Tool’.

STEP 4: DELIVER

To support safe use of the final solution, documentation and training must be created

for use during the deployment phase. The documentation would include the solution

specification ‘as built’ along with the following details:

● Instructions on using the AR system. ● Instructions on calibrating the system and recommended frequency of

calibration/updates. ● Any health and safety guidelines and warnings. ● Risk Assessment before use (this may be done by safety manager rather than

solution provider). ● Maintenance and cleaning instructions.

Prior to rolling out an AR solution, training for end users is also important. Training

delivered by the solution provider or the client (depending on contractual agreement),

should include:

• Introduction to the AR device and its functional use (including advantages and limitations).

• Duration of optimum time of use.

• Key safety risks such as distraction and eye strain.

• Findings on potential increased mental strain and distraction (or any other safety and human factor based on assessments/testing).

• Emphasis on technical and hardware limitations that may pose safety risks such as FOV, obstruction to peripheral view, etc.

14


www.theAREA.org


Study

For easing adoption of wearable AR displays, an approach incorporating a transition

period in which both current and AR-assisted methods are used allows operators to

become comfortable and confident in using the new technology on the workshop floor.

LOOKING AHEAD

Augmented Reality can be an effective tool to ensure that procedures are followed

consistently and accurately as part of quality management processes and in

compliance with industry regulations. In assembly use cases, AR delivers value by

presenting information digitally in context to the user to guide them through complex

assemblies. This case study has shown how the AR project cycle can be used for an

aerospace assembly use case. It demonstrates how a solution could be developed

with the usability and safety at the center of the design. The approach emphasizes the

topic of safety and usability, however, it should be noted that there are many other

functional and non-functional needs (e.g., UI/UX design, security, hardware software

agnosticism, etc.) requiring attention to create higher quality AR experiences across

enterprises.

The AREA is driving the development of best practices and policies that organizations

could use to govern AR safety and human factors assessments within the workplace.

Current AREA activities include management of a safety committee in which members

collaborate to highlight safety unknowns, identify and prioritize safety risks and

generate resources to educate and guide the members and the wider AR ecosystem

on the topic. Similarly, a human factors interest group within the AREA Research

Committee supports those members seeking to explore and discuss new approaches

to address the usability issues in design of AR systems.

This project has identified and engaged with companies, research institutes, private

and public safety organizations inside and outside the AREA member network. As the

AREA expands activities focusing on the safety agenda in AR, it will seek deeper

discussions and collaborate with all the enterprise AR ecosystem stakeholders to

reduce risk and increase usability of AR-assisted solutions.

this resource is made publicly available to anyone seeking to adopt … · area, to illustrate the...

Documents