usability results from human performance feedback .usability results from human performance ......

Download Usability Results from Human Performance Feedback .Usability Results from Human Performance ... Usability

Post on 23-Jul-2018

215 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • * RTI International is a trade name of Research Triangle Institute.

    Contract Number: GS-10F-009L7 RTI Project Number: 0209482.000.006

    Usability Results from Human Performance Feedback and Development (HPFD) and

    ePerformance System Users: Technical Report #1

    FINAL REPORT Submitted to

    Navy Human Performance Center (HPC N5)

    ATTN: CDR Mark J. Bourne, MSC, USN Dam Neck Annex

    2025 Tartar Ave, Suite 100 Virginia Beach, VA, 23461-1924

    Submitted by

    Elizabeth Dean, M.A.

    Kimberly R. Aspinwall, M.A. Michael J. Schwerin, Ph.D. Douglas E. Kendrick, M.S.

    RTI International*

    3040 Cornwallis Road Research Triangle Park, NC 27709-2194

    Telephone: (919) 316-3878

    Fax: (919) 541-1261 E-mail: schwerin@rti.org

    February 15, 2006

  • REPORT DOCUMENTATION PAGE Form Approved

    OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) 02-15-2006

    2. REPORT TYPETechnical Report

    3. DATES COVERED (From - To)January 2005 February 2006

    5a. CONTRACT NUMBER GS-10F-009L7

    5b. GRANT NUMBER

    4. TITLE AND SUBTITLE Usability Results from Human Performance Feedback and Development (HPFD) and ePerformance System Users: Technical Report #1

    5c. PROGRAM ELEMENT NUMBER

    5d. PROJECT NUMBER 0209482

    5e. TASK NUMBER 006

    6. AUTHOR(S) Elizabeth Dean, Kimberly Robbins, Michael Schwerin, Douglas E. Kendrick, Mark Bourne 5f. WORK UNIT NUMBER

    7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT

    NUMBER RTI International 3040 Cornwallis Road PO Box 12194 Research Triangle Park, NC 27709

    9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITORS ACRONYM(S) HPC-N5 11. SPONSOR/MONITORS REPORT NUMBER(S)

    Navy Human Performance Center Dam Neck Annex 2025 Tartar Ave, Suite 100 Virginia Beach, VA, 23461-1924 12. DISTRIBUTION / AVAILABILITY STATEMENT A Approved for public release; distribution is unlimited 13. SUPPLEMENTARY NOTES

    14. ABSTRACT The Navy Human Performance Center (HPC) and Navy Personnel Command (NPC) have reached the final phase of development for the Navys Human Performance Feedback and Development (HPFD) and ePerformance system. This second and final iteration of usability testing was designed to verify the functionality of system revisions made as a result of recommendations offered in Schwerin et al. (2004) and gather user perceptions of system usability and cultural issues that may affect HPFD and ePerformance system implementation. This report documents the results of usability test and user survey data collection. Usability timing and errors analyses suggest that system enhancements as a result of the first usability study improved the overall usability of the HPFD and ePerformance system, but users still have significant difficulty finding, opening and creating HPFD and ePerformance documents.

    15. SUBJECT TERMS HPFD, ePerformance, Navy, usability testing, Human Systems Integration (HSI)

    16. SECURITY CLASSIFICATION OF:

    17. LIMITATION OF ABSTRACT

    18. NUMBER OF PAGES

    19a. NAME OF RESPONSIBLE PERSON Michael Schwerin

    a. REPORT UNCLASSIFIED

    b. ABSTRACT UNCLASSIFIED

    c. THIS PAGE UNCLASSIFIED

    UNLIMITED 56 19b. TELEPHONE NUMBER (include area code) 919-316-3878

    Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39.18

  • i

    Table of Contents

    Executive Summary...................................................................................................................... 1

    1 Introduction....................................................................................................................... 3

    2 Literature Review ............................................................................................................. 5 2.1 Performance Appraisal Systems ............................................................................. 5 2.2 Usability Testing..................................................................................................... 7 2.3 Summary of Results from Usability Study I........................................................... 7

    3 Study Objectives................................................................................................................ 9

    4 Participants...................................................................................................................... 10 4.1 Iteration 1: Naval Meteorology and Oceanographic Center (NAVMETOCCEN)12 4.2 Iteration 2: USS HOWARD (DDG 83) ................................................................ 12 4.3 Iteration 3: Bureau of Naval Personnel (BUPERS).............................................. 12

    5 Instruments and Procedures.......................................................................................... 14 5.1 Usability Scenarios ............................................................................................... 14 5.2 Usability Survey.................................................................................................... 15 5.3 Analysis Methods.................................................................................................. 16

    6 Results .............................................................................................................................. 18 6.1 Participant Demographics..................................................................................... 18 6.2 Task Durations ...................................................................................................... 20 6.3. Usability Errors..................................................................................................... 26

    6.3.1 Total Error Frequency............................................................................... 26 6.3.2 Rate of Error Occurrence .......................................................................... 32 6.3.3 Most Frequently Occurring Error Type .................................................... 38

    6.4 User Ratings.......................................................................................................... 41

    7 Summary and Conclusions............................................................................................. 49 7.1 Key Findings......................................................................................................... 49

    7.1.1 HPFD System ........................................................................................... 49 7.1.2 ePerformance System................................................................................ 50 7.1.3 User Satisfaction ....................................................................................... 50 7.1.4 Suggestions for Further Improvement ...................................................... 50

    7.2 Limitations of Research ........................................................................................ 51 7.3 Future Research .................................................................................................... 51

    8 References........................................................................................................................ 53

    Appendix A: Usability Error Codes used for Usability Study I and Usability Study II ...... 56

  • ii

    List of Tables

    Table 1. Usability Test ParticipantsNumber of Users and Appraisal Tasks Completed ....................................................................................................................11

    Table 2. Demographic Characteristics of Usability Participants ...............................................19

    Table 3. Average Time to Complete HPFD Usability Testing Task by Task, Supervisor Status, and Location.....................................................................................................21

    Table 4. Average Time to Complete HPFD Usability Testing Task by Task and Study Period ...........................................................................................................................22

    Table 5. Average Time to Complete Usability Testing for ePerformance Tasks 1141 in Usability Study II, by Location................................................................................24

    Table 6. Total Error Frequency for HPFD by Task, Supervisor Status, and Location ..............27

    Table 7. Total Error Frequency for HPFD by Task and Study Period.......................................29

    Table 8. Total Error Frequency for ePerformance Tasks 11 through 41 in Usability Study II, by Task and Location....................................................................................30

    Table 9. Rate of Error Occurrence for HPFD by Task, Supervis

Recommended

View more >