research article measuring mobile ict literacy: short-message performance assessment in emergency...

16 IEEE TRANSACTIONS ON PROFESSIONAL COMMUNICATION, VOL. 56, NO. 1, MARCH 2013

Research Article

Measuring Mobile ICT Literacy: Short-Message PerformanceAssessment in Emergency Response Settings

—ELIZABETH AVERY GOMEZ AND NORBERT ELLIOT

Abstract—Research problem: A construct mediated in digital environments, information communication technology(ICT) literacy is operationally defined as the ability of individuals to participate effectively in transactions thatinvoke illocutionary action. This study investigates ICT literacy through a simulation designed to capture thatconstruct, to deploy the construct model to measure participant improvement of ICT literacy under experimentalconditions, and to estimate the potential for expanded model development. Research questions: How might amultidisciplinary literature review inform a model for ICT literacy? How might a simulation be designed that enablessufficient construct representation for modeling? How might prepost testing simulation be designed to investigatethe potential for improved command of ICT literacy? How might a regression model account for variance within themodel by the addition of affective elements to a cognitive model? Literature review: Existing conceptualizationsof the ICT communication environment demonstrate the need for a new communication model that is sensitive toshort text messaging demands in crisis communication settings. As a result of this prefect storm of limits requiringthe communicator to rely on critical thinking, awareness of context, and information integration, we designed acognitive-affective model informed by genre theory to capture the ICT construct: A sociocognitive ability that, at itsmost effective, facilitates illocutionary action—to confirm and warn, to advise and ask, and to thank and request—forspecific audiences of emergency responders. Methodology: A prepost design with practitioner subjectsallowed investigation of performance improvement on tasks demanding illocutionary action after training on tasks ofhigh, moderate, and low demand. Through a model based on the independent variables character count, wordcount,and decreased time on task as related to the dependent variable of an overall episode score , we were able toexamine the internal construct strength with and without the addition of affective independent variables. Resultsand discussion: Of the three prepost models used to study the impact of training, participants demonstratedstatistically significant improvement on episodes of high demand on all cognitive model variables. The addition ofaffective variables, such as attitudes toward text messaging, allowed increased model strength on tasks of highand moderate complexity. These findings suggest that an empirical basis for the construct of ICT literacy is possibleand that, under simulation conditions, practitioner improvement may be demonstrated. Practically, it appears thatit is possible to train emergency responders to improve their command of ICT literacy so that those most in needof humanitarian response during a crisis may receive it. Future research focusing on communication in digitalenvironments will undoubtedly extend these finding in terms of construct validation and deployment in crisis settings.

Index Terms—Crisis communication, mobile information communication technology (ICT) literacy, short-messageservices (SMS) text messaging, validation.

Given the multitude of available devices,features, and user interfaces, the measurement ofeffective performance of nonvoice communicationacross mobile technologies remains a challenge.Cell phones and personal-digital-assistant (PDA)palmtop computers were among the most affordableinformation communication technology (ICT)devices that are now bridging the connectivity gapat the root of the digital divide [1]–[4]. Allowingan alternate communication path in times of

Manuscript received January 03, 2011; revised August 31, 2011and May 09, 2012; accepted June 22, 2012. Date of publicationSeptember 24, 2012; date of current version February 19, 2013.E. A. Gomez is with the Avery Morgan InformationInnovators, LLC, Whippany, NJ 07981 USA (email:[email protected]).N. Elliot is with New Jersey Institute of Technology, UniversityHeights, Newark, NJ 07102 USA (email: [email protected]).Color versions of one or more of the figures in this paper areavailable online at http://ieeexplore.ieee.org.

IEEE 10.1109/TPC.2012.2208394

crisis, the resilient architecture of short-messageservices (SMS) bypasses the internet. Simply put,in crisis situations, SMS text messages get throughwhen voice communication fails. An SMS textmessage operates with minimal signal strengthand uses little battery power for processing. The160-character per message exchange limit ofan SMS text message offers an interoperablecommunication mechanism that has become anSOS equivalent when other communications fail[5], [6]. Because of its inexpensive implementationas an ICT device, SMS text messaging is viewedas a lowest-common denominator for nonvoicecommunication globally; as such, SMS use hasgrown exponentially around the globe for personaland professional use. The physical architecture ofSMS, through the use of application programminginterfaces (APIs), has also become a permanenttwo-way bridge with mobile services such asFacebook, Twitter, and global positioning systems[7], [8].

0361-1434/$31.00 © 2013 IEEE

GOMEZ AND ELLIOT: MEASURING MOBILE ICT LITERACY 17

Regarding the study of effective writtencommunication, however, the impact of constraintsof the 160-character per message designed byHillebrand and his colleagues in 1984 is largelyunknown [9]. A system to teach and evaluateeffective mobile ICT literacy, employed within SMScommunication, is needed. This need is especiallyapparent in crisis communication contexts. Theimportance of effective communication is bestunderstood in times of crisis when the exchangeof clear, accurate, and timely information is vital.The 2010 Haiti Earthquake’s dependency on SMStext messaging, for example, yielded approximately100,000 SMS messages, in total, from Mission4636, a text-message short code established within48 hours of the crisis [10], [11]. The short codewas made available for any individual in Haitito text, without cost, as a request for help. Only16,000 of the collected messages, however, weredecipherable; and only 3000 messages containedcontent sufficient to warrant action [12]. WhileICT literacy is clearly of critical importance inemergency relief situations, little is known aboutthe effective design of SMS text messages.

To provide base-line knowledge about effectiveSMS text messaging in emergency responsesituations, the present study was designed toassess the effectiveness of 300 short messagessubmitted by 50 participants within a controlledenvironment intended to simulate emergencyresponse conditions. The narrow range of the studyis justified to establish a program of researchintended to answer the following questions:

(1) How might a multidisciplinary literature reviewinform a model for ICT literacy?

(2) How might a simulation be designed thatenables sufficient construct representation formodeling?

(3) How might prepost testing simulation bedesigned to investigate the potential forimproved command of ICT literacy?

(4) How might a regression model account forvariance within the model by the addition ofaffective elements to a cognitive model?

If the construct of ICT mobile literacy can bedefined and measured, further research inconstruct definition and measurement may lead toenhanced training in effective SMS text messagingin emergency response contexts. In addition, themethodology presented in this study may itself bethe basis for an investigative strategy to examineknown constructs, such as writing, as they emergein digital environments.

This paper presents the literature review thatinformed our research, the methodology we usedto examine our hypotheses, and the results of ourstudy. This paper concludes with a statement ofthe limitations of our current work and suggestionsfor further research.

LITERATURE REVIEWAllowing conceptualization of communication indigital environments, a review of the literatureinvolving the construct of ICT literacy yieldsevidence-based justification and definition ofthe construct itself. As an investigation of thespecific communication behavior that this study isdesigned to measure, a literature review focusing onmultidisciplinary theoretical orientation, integrativeliterature selection, the ICT communicationenvironment, cognitive-affective modeling, andgenre theory provides a detailed explanation ofthe communication performance and a theoreticalorientation for future research.

Theoretical Orientation We define ICTliteracy within a cognitive-affective frameworkof performance as a construct that is enhancedby conceptualizations of genre. Fish [13] hasmaintained that interdiscliplinarity is impossibledue to the nature of stance—an epistemologicalmove toward a new line of investigational authorityresults only in additional singularity. We agreewith this limitation and further hold that the studyof theory-based explanations of performances indigital environments is best approached withina multidisciplinary framework that embracesresearch specializations that are complementary,even though they are not commonly associated. Asinvestigation continues on the ways that traditionalconstructs are mediated in digital environments,uncommon disciplinary associations becomenecessary.

Selection of Literature to Review As such,the following literature review is informed byspecialization identified by the National Center forEducational Statistics under the Classificationof Instructional Programs [14]: Professional,Technical, Business, and Scientific Writing (CIP09.0908 and 23.1303); Computer and InformationSciences (CIP 11.0101); Educational Assessment,Evaluation, and Research (CIP 13.06); and Rhetoricand Composition/Writing Studies (CIP 23.13).

ICT Communication Environment ICT,recognized as a new mode for communication,has seen exponential growth. In April 2010, a


TABLE IUSAGE OF NONVOICE ACTIVITIES ON A MOBILE DEVICE

Pew Internet and American Life Project pressrelease reported that 72% of cell-phone usersexercised text-messaging features on their phones[4]. On a typical day, usage from a mobile devicereveals that 72% are text messages, 34% are emailmessages, and 30% are instant messages—withSMS text-messaging more than 2 times greater thanthe use of email and instant messaging on a cellphone or PDA. The use of SMS text-messaging ofthese adults is 50% greater than all other nonvoiceactivities, with the exception of photography. AsTable I demonstrates, the percentage of nonvoiceactivities for cell phones or PDAs is rising rapidly.Rapid deployment of affordable mobile ICT, aswe have noted, has left little time to investigateexperimentally effective performance of SMStext messaging, a subset and constant nonvoicecommunication act that has emerged fromtechnical developments in the field of computer andinformation science.

Complementary to recognition of the rapid growthof SMS text messaging, the study of communicationemploying this communicative framework underspecific conditions, such as crisis communication,sharpens the critical components of mobile ICTliteracy. In initially formulating our concept ofmobile ICT literacy, we began with the definitionprovided by the American Library Association’sAssociation for Colleges and Research Libraries(ACRL). Within academic settings, informationliteracy is defined as a set of abilities requiringindividuals to “recognize when information isneeded and have the ability to locate, evaluate, anduse effectively the needed information” [15]. MobileICT literacy—a unique genre that problematizesthe ACRL academic definition in which librarydatabases are searched to produce discursive, often

lengthy student essays—is achieved within theICT architecture reliant on constrained characterlimit per message. Indeed, ICT literacy as usedin emergency response contexts is distinct fromsocial networking aims in which short answerswith multiple message exchanges allow formeaning to emerge under iterative conditions. In acrisis response mode, situational awareness andaccuracy are critical. Maximizing the 160-characterconstraint is key because the need for the back andforth dialogue could add time to respond; in times ofcrisis, signal and connectivity could subsequentlyfail as well if the initial message does not achievemaximum efficiency. Because of communicativeand technological challenges, the message must beexactly right the first time if it is to be effective.

Because the 160-character limit is a considerablelimitation to message formulation under anycondition, that constraint is exponentiallychallenging in emergency response setting inwhich responders (with a wide variety of mobileICT literacy ability) must communicate acrossthreatened geographic locations (that may expandover the duration of the critical incident). AsMcNair, Paretti, and Davitt note, “The richnessof context provided by spatial referents andparalinguistic indicators in co-located settingsis missing in distributed settings” [16, p. 235]where short-message use often occurs. Theseconstraints challenge the ability to convey a richcontext of situational awareness, communicativeaim, and message form. Part of the solution to thecharacter limit may be addressed by the real-worldapplication of plain language. Plain language [17]is an international movement that has been afederal mandate for government agencies datingback to 1995, with increased traction since 2004as agencies, such as the Federal EmergencyManagement Agency and the National Institutesof Health [18], have adopted principles of clarity,conciseness, and organization. Presently, the useof plain language is now a part of the Plain WritingAct of 2010 [19].

The association of plain language with simplecommunicative acts is nevertheless a risk, asHart-Davidson [20] from the field of Professional,Technical, Business, and Scientific Writing hasobserved. When writing is seen as a basic skillrather than an area of strategic activity impactingan entire organizational setting, poor choices result.It is therefore important to design the construct ofICT literacy to include both elements of charactercount and wordcount with message aim. Suchfundamental conceptualizations of communicative


exchange benefit by emphasizing action-orientedtasks that occur within specific episodes. Workingin the field of Rhetoric and Composition/WritingStudies, Lloyd-Jones explains the importance of adiscourse model to identify the creation of taskswhich stimulate writing in the appropriate range,but not beyond it, so that a communicative actmay be assessed [21]. As transactional models,speech acts performed in such defined discoursemodels are defined by Austin and Searle ascommunication with emphasis on illocutionaryand perlocutionary speech acts. In designing themessage form of action-oriented tasks occurringwith emergency response situations, we highlightin the present research the explicit nature of theillocutionary speech act, defined as performativeactions—to confirm and warn; to advise and ask;and to thank and request—when undertakingan obligation or commitment [22]. Emphasizingaudience needs, Searle notes that the successfulillocutionary act implies that the preparatoryconditions of the act—such as understandingaudience needs—are satisfied [23]. Accompanying aview of communication that embraces complexity,a transactional theory of discourse emphasizingthe aim of actions yields a defined environment inwhich the construct of ICT literacy may be studied.(For contemporary treatments of transactionalcommunication models in organizations, see Te’eni[24]–[26].)

On a mobile device, critical ICT literacy componentsneeded in crisis situations (to communicate exactlocation and evaluate explicit need) are furtherchallenged by device size, device capabilities, andconnectivity—a perfect storm of limits causing thecommunicator to rely heavily on critical thinkingdeployed in the service of an awareness of contextand an integration of information. Awareness ofcontext thus calls for a model of ICT literacy thatextends beyond a cognitive framework.

Cognitive-Affective Modeling A cognitive-affective orientation to performance, offered by bothMessick [27] and Mischel and Shoda [28], holdspromise for performance-based assessment. On onehand, developed abilities, such as communicatingin digital environments, may be understood to bea cognitive function; thus, measurement of thespecial case of SMS performance in an emergencyresponse situation consists of character count,wordcount, and performance time as they arerelated to an overall score on the performance. Onthe other hand, the addition of affective elements ofperformance, as Anastisi [29] long ago recognizedin the field of Educational Assessment, Evaluation,

and Research, contribute to measurement in theirability to reveal learners’ behaviors, attitudes,apprehension, enjoyment, and perception. A unifiedvision of performance, encompassing cognitive andnoncognitive variables, is important in that factors,such as perceptions of skill usefulness and tasktenacity, are part of the sociocognitive ecology wherelearning occurs and performance is measured.In professional settings, as Schmidt and Hunterhave demonstrated in their meta-analysis, affectivemeasures such as integrity tests (self reports usedto estimate conscientiousness, agreeableness, andemotional stability) provide a 20% improvementover purely cognitive ability measures in predictingtraining and job performance success [30]. (Forfurther analysis of the significance of affect, seeNobel Laureate psychologist Daniel Kahneman [31]on the role of emotion in risk perception.) In theirstudies of educational measurement, Willingham,Pollack, and Lewis [32] found that teacher surveysof student scholastic engagement, when added tostandardized test scores, increased the predictionof high school grade point average from 0.62 to0.90. Following Mischel and Shoda, we thereforeconceptualized ICT literacy as a performance thatis best understood if we learn more about theperformance itself and the way that learners viewthat performance. Yet, the concept of ICT literacyand its emergence within a cognitive-affective modelrequires a pragmatic concept of situated action ifwe are to fully understand the unique nature of ICTliteracy.

Genre Theory Communicative behavior, plainlanguage use, illocutionary action, and systemsof personality—operating together within a digitalenvironment—may ultimately be understoodas elements of a new, unique genre in the fieldof technical communication. While conceptsof transactional discourse, speech acts, andfederal law may be amorphous, these conceptsbecome fixed when they are viewed as ways ofunderstanding the new genre of mobile ICT literacy.

As Malone has demonstrated, genre has alwaysbeen a key area of study in historical studies oftechnical communication [33]. From studies oftechnical writing in Renaissance England [34] to thepublic representation of nanotechnology [35], genreresearch—often understood as studies of types ofdiscourse—reveals context. Miller has extendedthe structuralist nature of genre by defining itas a rhetorical action wedded to context [36].Because an understanding of genre is linked to thecircumstance, Miller argues for a pragmatic—ratherthan a syntactic or semantic—understanding of


situated actions. Since genre is also understoodas an element of social control, as Chandlerreminds us, the values and ideological assumptionsembedded in genre use are important to anunderstanding of any emergent form [37].

While much has been written about genre withincultural studies [38], disciplinary communication[39], technical communication [40], and writingassessment [41], recent research in genre theory asapplied to contemporary communication contextsallows an expanded horizon on which to viewmobile ICT literacy. As Luzón [42] has shown,genre research can improve our understandingof technical communication practice throughanalysis of the organizational and social contextsin which writing takes place. In the case ofmobile technology, Johnson-Eilola and Selber[43] identify context, change, content, and toolas a framework that emphasizes genre not as aninevitable response format but, rather, as a vehicleto achieve the best response to context and desiredchange. Similarly, Thatcher [44] offers a heuristicemphasizing purpose, audience, information,organization, and style in analyzing digital literacyacross contexts. The idea of fluidity across contextis key to Spinuzzi’s call for researchers to abandonthe dichotomy between observer and observed [45].To leverage our research on mobile and wirelesstechnologies, he believes that we should recruit ourparticipants into the research process with us. Inmediated environments, it seems, even the genre ofresearch is transformed toward multidisciplinaryends.

Viewed in terms of macroscopic, mesoscopic, andmicroscopic interrelated contexts, Spinuzzi [46]views genre as a vehicle revealing organizationalinteraction, goal-directed behavior within theorganization, and the minute practices of thoseworking to achieve specific goals. For Spinuzzi,genres are to be seen as traditions of artifactproduction, use, and interpretation, ideologies thatshape the very contexts in which they emerge.As symbolic representations of the developmentand stabilization of world views—including values,ethics, and related humanistic concerns—genrecan thus be traced in its movements within theecology of organizational settings. When mappedby techniques, such as activity system diagrams,genre analysis can tell us more about the dailypractice of workers and how they can be supportedto achieve their goals.

By studying genres in relation to the activitysystem of a community, researchers can explore

the communicative features of the repertoire ofgenres used by the community. Significant innew studies of genre within digital environmentenvironments is that the technology, as Bolterand Grusin argue, mediates communication[47]. While writing mediates cognition, as Ongfamously posited [48], digital environments alsomediate written transactions that are themselvesacts of mediated cognition. While they appearto be transparent, in fact, digital environmentsre-structure and re-present communicative actsso that use of nondiscursive symbolization relianton image becomes inherent to the communicativeexperience [49]. As Bolter and Gromala interpretthe nature of digital mediation, the transparencysuggested by the windows metaphor of MicrosoftCorporation is a myth [50]. The digital environmentis not there to capture; the digital environment isthere to transform. Within digital environments,the modernist orientation held by those who believethat transparency defines the communicativeexperience should be rejected; inherent in digitalenvironments are new ways that meaning may bemade [51], [52].

Such distinctions are beyond solely theoreticalinterest when we realize that much of what weknow in technical communication is based ondocuments that are static, linear, and text based.The explosion of digital media and use changesthat. Within nondiscursive digital environments,communication is cognitive as well as affective,thriving on connections that may be ephemeral [53].Within digital environments, where visual devicessuch as text-based emoticons and abbreviationsare employed to yield meaning, little is knownabout the construct of writing or, for that matter,if the word “writing” is even the correct term forsuch communicative transactions. Within digitalenvironments, writing is truly a mediated construct.

Carving the target domain of communication intoa manageable slice that emphasizes transactionswithin digital environments, then further whittlingdown that subset of the target domain, allows us toexamine the genre of mobile ICT literacy performedin emergency response situations.

Definition With an awareness of the traditionaldefinition of a construct—the postulated attributereflected in test performance [54], [55]—we offerthe following definition:

Within a digital environment of short messageexchange, effective mobile ICT literacy isdefined as the ability of individuals to


TABLE IIILLOCUTIONARY ACT MAPPING

participate effectively in transactions thatinvoke illocutionary action.

Based on a multidisciplinary review of ICT literacy,we propose the first study hypotheses:

Hypothesis 1 (H1). A multidisciplinaryapproach to ICT literacy will result in adefinition that is sufficiently capacious tocapture the construct under investigation.

METHODOLOGY

In this study, we designed our methodologyto capture the complexities of the constructunder examination. As such, we employedevidence-centered design (ECD), an approachto constructing research that allows evidentiaryarguments to be built into the research fromthe design stage [56]–[58]. Designing researchwithin such a framework allows appropriatechoice of methodologies, selection of constructedresponse tasks, identification of subjects, anddefined research and reporting protocols. AnECD approach is essential for research in theemergency-management domain where eachincident has unique situational dimensions. ECDis evidentiary in orientation, enabling data analysisto be performed that addresses validation andgeneralizability issues associated with a program ofresearch leading to further study.

Choice of Research Methodology Our studyused a mixed-modes research methodology witha repeated measures design. To execute thebaseline study of the genre of mobile ICT literacy,we designed a quantitative study that wouldallow descriptive and inferential analysis of theconstruct under emergency response simulations.The cognitive-affective framework we employed

allowed the measurement of performance and thesubject’s view of that performance. Participantswere given background information on the study,assessed (pretraining), trained, and then assessed(posttraining) in order to determine baselineperformance information and the possibility ofperformance improvement. Because our researchwas undertaken in order to document a newconstruct, a quantitative method was needed for thestatistical analysis associated with experimentalresearch. Adoption of qualitative research, suchas ethnographic study and critical methodologies,such as discourse analysis, will add considerablyto future research about mobile ICT literacy inemergency response and other contexts.

Experimental Tasks To concretize the constructof mobile ICT literacy, tasks (or assessmentepisodes yielding a constructed response) werebased on the illocutionary framework specifiedbefore. Participants were given an audio/videosimulation of a crisis scenario in pairedapplications. That is, participants were given theseries of paired tasks shown in Table II—to confirmand warn, to ask and advise, and to thank andrequest—before and after training. Emphasizingan assessment episode, Ruth and Murphy’sassessment conceptualization—addressing theimportance of the task prompt as it occurs withina defined context—was fundamental to our designof each task [59].

As a crisis scenario, the tasks provided anarrative structure for the prepost-test design [2].Participants took on the role of a team volunteerasked to communicate with a coordinator managingthe relief effort. In the task, participants wererequired to provide assistance for an imaginedorganization, named Independence for Seniors andPeople with Disabilities, located in Northern New


Jersey. Under conditions of severe weather thatincludes power outages and flooding, the taskdemanded that the participant work to ensure thatboth seniors living alone and those with disabilitieshave resources (defined as food, flashlight, andbattery-operated radio) at their reach. The scenariobegins as torrential rain and flooding begins andthe early warning notification is given.

Under the persona of the team volunteer, theparticipant communicates with the managementcoordinator. As the participant stops at each homeon a given list, the SMS message is written tothe management coordinator to confirm that theresident has the identified resources at hand. Aspart of the task, participants find that residents donot have these resources, and plans must be madeto access supplies stored in a central location.Because these are short messages that must bewritten rapidly under conditions where multiplemessages may not be an option, response time iscritical. Our study revealed average response timeswere under three minutes. The simulation designled to the second hypothesis of the study:

Hypothesis 2 (H2). A simulation can bedesigned that will allow sufficiently robustconstruct representation of ICT literacy.

Subjects The study was designed to recruit abalance of men and women who were currentor past emergency response volunteers in theircommunity. Because of the complexities of analysis,we sought to recruit as few participants as possible,while maintaining a sample size ( 35) that wouldallow the use of inferential analysis. Recruitmentof a comparison group was also prohibitive due toresource constraints and a lack of knowledge ofthe construct itself. Participants were recruited byemail and phone calls from emergency responseorganizations known to the researchers and theircolleagues.

The 50 volunteer participants actually selectedfrom the purposive sampling plan were current orpast crisis responders. A survey of the selectedparticipants revealed the following categories ofidentification: community volunteer 15),practitioner with crisis response abilities ( 16),academic researcher ( 2), student ( 4), none( 10), other ( 3). Participants were selectedbased on their practitioner role or volunteerroles in communities. While the participants metthe selection criteria of the study, they did notself-select a designated crisis response role (hencethe designation as none and other). The samplingplan contained an even gender division (25 female

Fig. 1. Web-based training application for mobile ICTliteracy.

and 25 male). The age range was as follows: 22%11) were 18–34; 36% ( 18) were 35–49;

34% ( 17) were 50–64; and 8% ( 4) over 65.Based on tasks designed to elicit six messages perparticipant, we collected 300 written responsesfrom the participants. To ensure protection of theparticipants, the study was conducted under theInstitutional Review Board (IRB) Protocol NumberE80-07.

How the Study was Conducted The study wasexecuted through a web-based platform whichthe participants could access at their desktops,a design allowing us to reach participants fromsix states and two countries. This method allowedconsistent delivery of training, data analysiswithin the simulation platform, and reduction ofdevice-related variables. Because our focus is onmessage form and content, the use of the deviceitself was not studied and instead noted as an areafor further research. The final web-based platformtraining application shown in Fig. 1 was developedfollowing the results of usability protocols followedin the pilot test. Tasks were designed to simulatetransactions key stroked on a handheld device.Participants had to perform action-oriented writtencommunication within the 160-character limit.

Following best practices of usability testing[60], the study began with a pilot test of fourparticipants (ages 36 to 64, three women, and onemale), who were asked to assume the role of acommunity responder. The purpose of the pilottest was to ensure successful integration of theentire training process, and test the flow betweenindividual components. The computer-based,multimedia simulation was recorded in Camtasiato allow analysis of audio comments, screen search


patterns, and key stroking. Particular attentionwas given to the ability of participants to navigatethe simulation and their response style patterns(pauses, backspacing, and rewording responsemessages). While this analysis was qualitative innature, a pretraining survey of 38 questions anda posttraining survey of 62 questions were alsoused. Questions were associated with the training,as well as with the attitudes of the participantstoward SMS use. The pretesting phase, lasting15 weeks, led us to changes in the computersimulation to allow ease of use and refinement ofthe questionnaires we would use to capture theaffective components of mobile ICT literacy.

Following the pilot, the study was conducted inthe following steps on the web-based platform:

(1) Participants logged into the simulation andcompleted informed consent.

(2) Participants viewed a multimedia presentationwith background information on the study andthe use of plain language before the trainingbegan.

(3) Participants were given the pretraining survey,with questions designed to tap affectivebehavior.

(4) Participants were given three pretrainingtasks to elicit performance-based variables(character count, wordcount, performancetime) recorded in the database.

(5) Participants were then provided with threedistinct training modules on effective mobileICT literacy. Each training module was pairedwith a posttraining task with emphasis onwriting effective SMS messages. Identicalperformance-based variables (character count,wordcount, performance time) for each taskwere recorded in the database.

(6) Participants were given the posttrainingsurveys, with questions designed to tapaffective behavior.

(7) The simulation ended, and participants werethanked.

The seven-step assessment took 30 to 40 minutes,and all communication with the participants tookplace within the computer-mediated environment.Task and survey data were collected within aMySQL database designed for the simulation. Theprepost study design led to the formation of thethird hypothesis:

Hypothesis 3 (H3). A computer-based,multimedia simulation will allow prepost test

investigation of improved performance of ICTliteracy.

Data Analysis Following the ECD framework of thestudy, design of the MySQL database accompanyingthe seven-step assessment allowed cognitive andaffective variables to be recorded and displayedin Excel format. As participants keystroked taskresponses onto the simulated handheld device inFig. 1, character count, wordcount, and time ontask were systematically recorded using a simplecount of each of these independent variables. Eachof the scorable training modules was designed toincorporate four goals appropriate to the genreof mobile ICT literacy: (1) decrease of charactercount; (2) decrease of wordiness; (3) decrease oftime on task; and (4) overall command of episodeproficiency (the episode score) in response to theillocutionary acts (to confirm and warn, to adviseand ask, and to thank and request).

Informed by the definition of mobile ICT literacyoffered before, we defined the cognitive aspects ofthe model as character count, wordcount, and timeon task as independent variables that would berelated to the dependent variable , the overallepisode score. The need for relational data analysisof the independent-dependent variable model led tothe final study hypothesis.

Hypothesis 4 (H4). A model of ICT literacyshould combine the independent variablesof character count, wordcount, and time ontask as related to the dependent variableof an overall episode score . By means ofregression analysis, that operational shouldbe able to be examined with and without theaddition of affective independent variables.

The ability to respond accurately to an episodeat established levels, the dependent variable ,was measured by having experts use a four-pointscale with rubric, shown in Table III, to estimatehow well the study participant performed on theepisode. Carefully selected, the experts were twoemergency response specialists currently assignedto Office of Emergency Management initiatives;both were active with Homeland Security initiativesand both had an extensive history of involvementwith emergency response, community initiatives,and training. After we reviewed the rubric with thespecialists, we asked them to score the responsesand to justify the reasons for their responses.Tapes of the scoring sessions revealed that therubric allowed a firm basis for the episode score.For instance, for a low–medium response (ascore of 2), the experts noted that the response


TABLE IIIRUBRIC USED BY EMERGENCY RESPONSE SPECIALISTS TO AWARD THE EPISODE SCORE

contained information, but for the most part thatinformation, while coherent, was not usable. For amedium–high response (a score of 3), the expertsnoticed “a dialogue starting to develop.” That is,the response was clearer and contained someactionable (situational awareness) content. For ahigh response (a score of 4), the experts confirmedthat nothing was wrong with the answers butthat, in some cases, additional, situation-orientedinformation would have been helpful confirmation.Across all response levels, the experts confirmedthat the responder role—how the sender wasable to provide succinct, situation-orientedinformation—impacted the expectations of theresponse and, thus, influenced the response score.Our method of obtaining an episode score fromexperts confirms that while rater training in theuse of a rating scale does not necessarily eliminatevariability in score attribution, questioning ratersabout the reasons for their decisions for theirscoring allows a fuller understanding of how rubricsare used by raters to award scores [61].

The affective variables were incorporated into thepretraining surveys, with Likert-scale questionsdesigned to evaluate participant perception of ICTdesktop behavior, ICT mobile usage behavoir,text-messaging attitudes, writing behavior, writingapprehension, and writing enjoyment. In general,these questions were informed by concepts ofindividual usage behavior that would yield amore complete understanding of self-confidenceof the ability of participants to perform thetasks at hand [62]. The second set of affectivevariables was taken from posttraining surveys,with Likert-scale perception questions based onusefulness, performance, learning, readiness,intent, enjoyment, motivation, preparedness, andwriting behavior. These questions were informedby studies on computer-mediated communicationthat would tell us more about the participants’perceptions of learning and their ability to achieveproficiency in mobile ICT literacy [63]. Both surveys

were built on the belief that the perceptionsof responders toward mobile ICT literacy are asignificant cornerstone of crisis response readiness.The affective variables from the surveys wereanalyzed through factor analysis with a Varimaxrotation. Factor loadings on the pretest rangedfrom 0.33 to 0.86 and from 0.42 to 0.94 on theposttest. On the pretest, the questions accountedfor 73.56% of the variance; on the posttest, thequestions accounted for 77.45% of the variance.While the cognitive aspects of the model wereidentical, the questions involved in the affectiveaspects of the model were not. This design wasused to improve model strength based on usagebehavior during the pretest phase of the studyand performance perception during the posttestphase of the study. As such, the repeated measuresdesign is an accurate description of the cognitiveaspects of the model.

Performance and survey data were uploaded toPSAW 18.0 statistical software for further analysis.Both parametric and nonparametric analyseswere used to establish construct representation,improved performance, and regression modeling ofH2, H3, and H4, respectively. A 95% confidenceinterval was used for all significance testing. Tocomply with the IRB and privacy, no identifiabledata for the study participants were stored.

Ensuring Validity and Generalizability In ourattempt to map a new construct and demonstratea way to assess its dimensions, we engaged inan extensive literature review of contemporaryviews on validity from the field of EducationalAssessment, Evaluation, and Research.

With Messick [64], we understand validity as

an integrated evaluative judgment of thedegree to which empirical evidence andtheoretical rationales support the adequacyand appropriateness of inferences and actions


TABLE IVPAIRED -test

based on test scores and other forms ofassessment. (p. 13)

While this definition was offered in 1989 tosupport research in the field of educationalmeasurement, it is applicable to the kind ofresearch we are reporting. Employing statisticalmeasures to establish confidence intervals,taping experts as they awarded overall episodescores—these quantitative and qualitative methodsare ultimately never stamps of validity but, rather,stages in a process of validation. At best, suchstrategies—especially when they are deployedin support of a new construct—are meant asguideposts to help readers judge the adequacy ofconstruct inferences and the potential for futureresearch.

Viewed in such a fashion, the concept ofgeneralization is inextricably linked to validation.The ability to generalize based on evidence is avery difficult assurance when language use is onthe table in an assessment of performance. Thebackground of the writer, the complexity of thetask, the method of assessment—even the structureof reporting—are all sources of variance that aredifficult to control in the present and even morechallenging to generalize for the future. Recentadvances in Validity theory by Borsboom [65] haveadvanced variation as a measure of validity. Thatis, if variation in the attribute causes variation inscores, then the assessment has been validated.This definition of validity is especially appealingin the present study because of the variationsin illocutionary acts undertaken in pretests andposttests. If variation is observed—in statistically

significant differences in scores and in variance inregression models—then the reported study maybe judged as both valid and able to be generalizedto other settings. The emphasis on evidence isnot abandoned under such a model; rather, it isthe assembly line metaphor that is shelved. Assuch, we propose that if H3 and H4 are partiallysupported in their ability to reveal score variation,then future studies will be possible to furthermap the construct—if those studies themselvesare designed to allow for variance. As Borsboomnotes, a view of validity as variation focuses onconstruct representation, with nomothetic spantaking second place to idiographic representation.

RESULTSFollowing the definition of validity as an integratedjudgment yielding observed variation, we now reportresults for our baseline study. Tests of significancedetermine improvement; regression analysesdetermine the strength of the cognitive-affectivemodel employed in the research.

As Table IV shows, participant performance onEpisodes 1 and 4, judged to be of moderatecomplexity, demonstrated the desired statisticallysignificant decrease in character count andwordcount after training. Time on task and episodescore, however, were not statistically significant.The performance on Episodes 2 and 5, judged to beof high complexity, showed statistically significantimprovement across all measures. Participantresponses on Episodes 3 and 6, determinedto be of low complexity, showed statisticallysignificant differences on time on task and episode


TABLE VLINEAR REGRESSION MODEL FOR MOBILE ICT LITERACY


score. Participants took longer to complete thetask at statistically significant levels; however,this measure we interpret as problematic whendecreased time on a task was desirable. Althoughcharacter count and wordcount did decrease on thelow complexity task, statistically significant levelswere not recorded.

Overall, the results confirm H2 and H3: A baselineof ICT literacy was able to be measured in asimulation that revealed performance improvement.However, we were not able to confirm that astatistically significant level of performance wasachieved on all variables of all tasks after training.Only on tasks of high complexity did participantsdemonstrate statistically significant improvementon all variables; on measurement of improvementon tasks of low and moderate complexity,challenges remain. It appears that performancenuances present in nonextreme situations mighthave been required that were not captured in themodel. The design of the tasks of moderate andlow complexity themselves, for instance, may havecontributed to unanticipated sources of error. Aswell, it may be possible that only the most extremecases require that character count, wordcount, andtime on task must be decreased to improve theoverall episode score.

Hints of such nuances appeared when the episodescores were given. Our two field experts notedthe importance of situational awareness in amessage for improved response efforts; our model,however, did not assess the presence of situationalawareness in the written response, and that isperhaps one of the independent variables absentfrom the evaluative model. Hence, H3 cannotbe confirmed in its entirety. While improvedperformance of mobile ICT literacy was able to bemeasured, one or more additional variables mayhave intervened as extraneous variables that, alongwith task design, impacted statistical significance.

Linear regression, as shown in Table V, was used togenerate an operational model that would provideinformation on the cognitive domain alone and incombination with the affective domain. In partialconfirmation of H4, measurement of the cognitiveaspect of mobile ICT literacy, character count,wordcount, and time on task could be used topredict the overall score across all three tasks.In the pretests and posttests for episodes 1 and4, tasks of moderate demand, regression for thecognitive elements captured 28% and 20% of thevariance, respectively. In the pretests and posttestsfor episodes 2 and 5, tasks of high demand,

regression for the cognitive elements captured29% and 32% of the variance, respectively. Inthe pretests and posttests for episodes 3 and 6,tasks of low demand, regression for the cognitiveelements captured 35% and 18% of the variance,respectively.

When the cognitive variables were combinedwith the affective variables for episodes 1 and 4,however, statistical significance was producedonly for the pretest, with the model accountingfor 33% of the variance. In the pre and posttestsfor episodes 2 and 5, the combination of cognitiveand affective variables accounted at statisticallysignificant levels for 39% and 48% of the variance,respectively. In the pretests and posttests forepisodes 3 and 6, the combination accounted atstatistically significant levels for 46% and 42%of the variance, respectively. As was the case inthe study of improved performance, tasks of highcomplexity yielded the clearest study results.

The combination of cognitive and affective variablesallowed a statistically significant regression modelto be built that accounted for increasingly higherlevels of construct representation from pretestto posttest. However, complete confirmationof H4 remains problematic. While statisticallysignificant levels for the regression analysis wereevidenced for episodes 1 and 4, the cognitivemodel actually decreased from pretest to posttest,and the combined model failed to meet levels ofstatistical significance in the posttest. Further,the pretest cognitive scores, while statisticallysignificant for episodes 3 and 6, actually decreasedin model strength from pretest to posttest onthis low-demand pair of tasks. And, while thedecrease was slight, the regression data fromthe combined model of cognitive and affectivevariables also decreased in these paired tasks.As was the case with the performance measures,the internal consistency measures held only ontasks of high complexity. Because performancenuances present in situations of low-to-moderatecomplexity appear not to have been captured in themodel, here is further confirmation that replicationstudies, with additional variables included toincrease performance scores and variable strength,are needed. As such, only partial confirmationcan be concluded for H2: In the present study,only the simulation of extreme crisis situationscan be designed that will allow robust constructrepresentation of ICT literacy as it is currentlyconceptualized.


CONCLUSIONS, LIMITATIONS, AND SUGGESTIONSFOR FURTHER RESEARCH

The use of qualifiers to confirm a hypothesissuggests the complexity of mobile ICT literacy.Construct definition and the environment forplanned emergence are difficult to confirmabsolutely, as are the measures used to estimateits presence. Indeed, we might wonder if we are,in reality, confirming the ideographic natureof performance assessment as we offer ourconclusions, study limits, and directions for furtherresearch.

Conclusions Of equal significance to the presentstudy itself, we have presented an approach tovalidate new communication constructs thatemerge in digital environments. Aspects of thetarget domain of communication may be so deeplymediated within digital environments that newconstructs, such as mobile ICT literacy, emerge thatare related to, yet distinct from, fully researchedconstructs evidenced in known genres. At facevalue, a written proposal may be more complexto define and evaluate than an SMS emergencyresponse message of 160 characters; nevertheless,as the present study has demonstrated, thevariables of mobile ICT literacy are difficult tocapture, and they may shift according to task. Thecomplexity of measuring forms of literacy in digitalenvironments, as well as interpreting complexfindings from experimental studies, suggests theneed to confirm the multidisciplinary approachassumed in H1.

Ultimately, the ECD model of validation used inthis study yields four advantages. First, the studyis designed in terms of what can, and cannot,be reported. While a regression coefficient is theresult of the variable model we designed, forexample, the model is a preliminary, not a final,confirmation of H2 and H4. More needs to beknown about the meaning of the episode score,for example, if the model is to be expanded. And,as we will explain, the model we have presented isnot as deeply contextualized as it appears. Hence,the second advantage is apparent: In emerging,contingent environments of digital communication,research is perpetual. There is always a contingentquality to reported results because the validityargument itself is a rhetorical formulation ofexisting evidence, not a final statement of proof.Within digital environments where technologiescontinue to emerge and mediate constructs,what Kane refers to as the contingent characterof validation is especially important [55]. Third,

because the rhetorical and contingent characterof the validation argument is acknowledged,the researcher is forced to be explicit about theconstruct under examination. Definition of thecommunication performance to be measured isthus viewed as a demand for construct validity.In our case, the definition of mobile ICT literacy(an ability of individuals to participate, withindigital environments, in transactions invokingillocutionary action) allowed, in turn, the design ofthe pre and posttests so that they would promptsuch actions (to confirm and warn, to advise andask, and to thank and request). Once the actswere designed, a variable model was then logicallydeveloped to capture the relationship between thecognitive and affective independent variables tothe dependent variable. This sense of precisionyields the final benefit of an ECD model: modeluse. In the case of crisis responders, research ofICT literacy models should be viewed, ultimately, ascontributing to life-saving measures. Because crisiscommunication for public health and emergencymanagement informs shareholders of events inwhich human lives are at stake, the need foreffective two-way interoperable communicationremains paramount. In many cases, SMS is alifeline during times of crisis as revealed in thecoordinated attacks against the US on September11, 2001; the Indian Ocean tsunami in 2004;Hurricane Katrina in 2005; the Virginia Techshootings in 2007; the earthquakes in Haiti andChile in 2010; the Pakistan floods in 2010; and the2011 Norway attack.

Limitations So that a performance standardcan be designed to be used in crisis situationswhen ICT resources are of vital importance, theinformation collected in our study suggests theneed for further study of mobile ICT literacy. Wehave attempted to highlight the limitations of theresearch as we have presented it: the narrowslice of the target domain under investigation,the limited tasks based on illocutionary action,the small sample size and lack of comparisongroup, the constraints of the construct due tocomputer simulation, a hesitancy in absoluteconfirmation of each hypothesis, and the presenceof ideographic representation of the constructthat may interfere with nomothetic generalizationsneeded to predict effective construct use acrosstime and circumstance. Our closing section focuseson future research design considerations in pursuitof mobile ICT literacy.

Suggestions for Further Research The mobile ICTliteracy model offered in this study shows promise


for designing text-based exchange protocols thataim to increase e-readiness for crisis response. It isequally clear that more needs to be known aboutthe variables and contexts of mobile ICT literacy ifa programmatic approach is to be offered that trulyserves those who work in crisis environments.

First, more needs to be known about the variablesof word choice, fluency, and rhetorical stance inconstruct representation and modeling, the essenceof this study. Advanced representations of thewriting construct such as that offered by Deane[66] categorize skill by social, conceptual, textual,verbal, and lexical/orthographic representation.Based on print-based views of writing, such modelsare characterized by hierarchy: Social skills suchas the ability to engage readers are tacitly assumedto be more complex than the lexical/orthographicskills associated with standard written English.In cases of ICT literacy performances in crisiscommunication transactions enacted in digitalenvironments, however, there appears to be greatskill in constrained language use—in messages thatrhetorically construct a defined role for a readerto play. While our study relied on count in termsof capturing the independent variables, furtherethnographic and discourse research should focuson quality of word choice, control of fluency, andrhetorical stance of the responder. The need forsuch study is indicated by the tape recordingsfrom our research. As the tape recordings revealed,perception of the responder role—how the senderwas able to provide distinct, context-orientedinformation under conditions of stress—is anidentified, yet not fully understood, variable.We might well wonder whether the regressioncoefficient might be strengthened if the variableof responder role could be defined, added, andevaluated as one of the independent variables ofthe study.

Second, more needs to be known about therelationship of the individual response to theecology of the crisis. As recent literacy assessmentframeworks such as those proposed by Wardleand Roozen [67] suggest, an ecological perspectiveallows attention to be given to the contextualnature of discourse. In their study of Mercy Corps,an international relief organization, Smith and

Young identify the need for the creation of alearning-oriented culture that takes into accountthe embedded nature of transactions withinorganizations operating in multinational settings[68]. As much as technology can help to lessengeographic isolation and provide timely accessto key resources, Smith and Young found, inagreement with Pacey [69], that the presence oftechnology alone does not guarantee successfulcommunicative transactions. Organization-widetechnology initiatives need to be augmentedwith initiatives that focus on people and driveorganizational conversations around learningand development. While our research centers onthe individual transactional level, more needs tobe known about the use of mobile ICT literacywithin organizations and the environments theyserve. While SMS use within a mobile ICT literacyframework provides a baseline across platformsused for global communication acts that holdthe potential to provide immediate relief, projectand organizational maturity are challenges thatare magnified in crisis response due to theuncertainty—in severity, location, timeliness—ofthe crisis. The longitudinal impact of messages,such as those elicited in our study, must be betterunderstood in their relationship to postcrisisplanning; that is, the individual message must notbe viewed as a homeostatic event but, rather, asinformation integral to geographical mapping, forexample, of an emerging public-health crisis. Asthe Institute of Medicine has observed, the need forprecise understanding of the organizational aspectsof technology practice becomes especially critical asa crisis episode recedes and public-health issues,such as cholera in Haiti, emerge [70], [71].

At the time of this writing, it is clear that effectivetextual information provided in short bursts duringa crisis will remain a common denominator inemergency-management situations. The extendeduse of mobile devices will surely keep textcommunication at the forefront for informationsharing. The presence of mobile ICT literacy withinsuch contexts reveals the need for further empiricalresearch—and timely training based on thatresearch—in the service of humanitarian response.

REFERENCES

[1] E. Avery Gomez. (2008). Connecting communities of need with public health: Can SMS text-messagingimprove outreach communication? Proc. 41st Hawaii Int. Conf. Syst. Sci. [Online]. Available:http://www.computer.org/portal/web/csdl/doi/10.1109/HICSS.2008.99


[2] E. Avery Gomez and M. Turoff. (2007). Community crisis response teams: Leveraging localresources through ICT E-Readiness. Proc. 40th Hawaii Int. Conf. Syst. Sci. [Online]. Available:www.computer.org/portal/web/csdl/doi/10.1109/HICSS.2007.135

[3] S. Fox and G. Livingston. (2007). Hispanics with lower levels of education and Englishproficiency remain largely disconnected from the Internet. Pew Internet. [Online]. Available:http://www.pewinternet.org/pdfs/Latinos_Online_March_14_2007.pdf

[4] A. Lenhart. (2010). Cell phones and American adults. Pew Internet. [Online]. Available:http://www.pewinternet.org/Press-Releases/2010/Cell-phones-and-American-adults.aspx

[5] T. Kass-Hout and X. Zhuang, Eds., Biosurveillance: Methods and Case Studies. Boca Raton, FL: CRC, 2010.[6] Information for Development Program of World Bank, the United Nations ICT Task Force, and the Wireless

Internet Institute (W2i), a Division of World Times, Inc. The Wireless Internet Opportunity for DevelopingCountries. [Online]. Available: http://www.infodev.org/en/Publication.24.html

[7] C. E. Ortiz. The Wireless Messaging API. [Online]. Available: http://developers.sun.com/mobility/midp/articles/wma/

[8] Java Community Process. Wireless Messaging API. [Online]. Available: http://www.jcp.org/en/jsr/detail?id=120

[9] F. Hillebrand, F. Trosby, K. Holley, and I. Harris, Short Message Service (SMS): The Creation of PersonalGlobal Text Messaging. Hoboken, NJ: Wiley, 2010.

[10] V. Hester. CrowdFlower impacts the future of disaster relief. Haiti Rewired. [Online]. Available:http://haitirewired.wired.com/profiles/blogs/crowdflower-impacts-the-future

[11] K. Heppler. Helping Haitians via mobile, crowdsourcing & social media. Social Brite. [Online]. Available:http://www.socialbrite.org/2010/02/16/helping-haitians-via-mobile-crowdsourcing-social-media/

[12] CNN iReport. Looking for Loved Ones in Haiti. [Online]. Available: http://ireport.cnn.com/ir-topic-stories.jspa?topicId=381628

[13] S. Fish, “Being interdisciplinary is so very hard to do,” Profession, pp. 15–22, 1989.[14] National Center for Educational Statistics. Classification of Instructional Programs. [Online]. Available:

http://nces.ed.gov/ipeds/cipcode[15] Association for Colleges and Research Libraries. Information Literacy Competency Standards

for Higher Education. [Online]. Available: http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm

[16] L. D. McNair, M. C. Paretti, and M. Davitt, “Towards a pedagogy of relational space and trust: Analyzingdistributed collaboration using discourse and speech act analysis,” IEEE Trans. Prof. Commun., vol. 53,no. 3, pp. 233–248, Sep. 2010.

[17] Plain Language.gov. Improving Communication From the Federal Government to the Public. [Online]. Available:http://www.plainlanguage.gov/

[18] Federal Emergency Management Agency. (2005, Sep.) FEMA Promotes Plain Language Radio Response.[Online]. Available: http://www.impact-information.com/impactinfo/newsletter/plwork18.htm

[19] Plain Writing Act of 2010. H. R. 496. [Online]. Available: http://www.govtrack.us/congress/bill.xpd?bill=h111-946

[20] W. Hart-Davidson, “Content management: Beyond single source,” in Digital Literacy for TechnicalCommunication: 21st Century Theory and Practice, R. Spikla, Ed. New York: Routledge, 2010, pp. 128–143.

[21] R. Lloyd-Jones, “Primary trait scoring,” in Evaluating Writing: Describing, Measuring, Judging, C. Cooper andL. Odell, Eds. Urbana, IL: National Council of Teachers of English, 1977, pp. 33–66.

[22] J. L. Austin, How to Do Things With Words. Cambridge, MA: Harvard Univ. Press, 1962.[23] J. R. Searle, Speech Acts: An Essay in the Philosophy of Language. Cambridge, UK: Cambridge Univ. Press,

1969.[24] D. Te’eni, “Review: A cognitive effective model of organizational communication for designing IT,” MIS Quart.,

vol. 25, no. 2, pp. 2–62, 2001.[25] D. Te’eni, “The language-action perspective as a basis for communication support systems,” Commun. ACM,

vol. 49, no. 5, pp. 65–70, 2006.[26] D. Te’eni, A. Sagie, D. G. Schwartz, N. Zaidman, and Y. Amichai-Hamburger, “The process of organizational

communication: A model and field study,” IEEE Trans. Prof. Commun., vol. 44, no. 1, pp. 6–20, Mar. 2001.[27] S. Messick, “Potential uses of noncognitive measurement in education,” J. Educ. Psychol., vol. 71, no. 3,

pp. 281–292.[28] W. Mischel and Y. Shoda, “A cognitive-affective system theory of personality: Reconceptualizing situations,

dispositions, dynamics, and invariance in personality structures,” Psychol. Rev., vol. 103, no. 2, pp. 246–268,1995.

[29] A. Anastasi, “A century of psychological testing: Origins, problems, progress,” in Exploring Applied Psychology:Origins and Critical Analyses, T. K. Fagen and G. R. VandenBos, Eds. Washington, DC: Amer. Psychol.Assoc., 1993, pp. 13–36.

[30] F. L. Schmitt and J. E. Hunter, “The validity and utility of selection methods in personnel psychology,”Psychol. Bull., vol. 124, no. 2, pp. 262–274, 1989.

[31] D. Kahenman, Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.[32] W. W. Willingham, J. M. Polack, and C. Lewis, “Grades and test scores: Accounting for observed difference,”

J. Educ. Measure., vol. 39, no. 1, pp. 1–37, 2002.


[33] E. A. Malone, “Historical studies of technical communication in the United States and England: A fifteen-yearretrospection and guide to resources,” IEEE Trans. Prof. Commun., vol. 50, no. 4, pp. 333–351, Dec. 2010.

[34] E. Tebeaux, The Emergence of a Tradition: Technical Writing in the English Renaissance, 1475-1640. Amityville,NY: Baywood, 1997.

[35] B. Faber, “Popularizing nanoscience: The public rhetoric of nanotechnology, 1986–1999,” Tech. Commun.Quart., vol. 15, no. 2, pp. 141–169, 2006.

[36] C. R. Miller, “Genre as a social action,” Quart. J. Speech, vol. 70, no. 2, pp. 151–167, 1984.[37] D. Chandler. An Introduction to Genre Theory. [Online]. Available: http://www.aber.ac.uk/media/

Documents/intgenre/chandler_genre_theory.pdf[38] S. Ngai, “Our aesthetic categories,” PMLA, vol. 125, no. 4, pp. 948–958, 2010.[39] C. Berkenkotter and T. Huckin, Genre Knowledge in Disciplinary Communication: Cognition/Culture/Power.

Hillsdale, NJ: Earlbaum, 1995.[40] C. Bazerman, “Introduction: Changing regularities of genre,” IEEE Trans. Prof. Commun., vol. 42, no. 1, pp.

1–2, Mar. 1999.[41] S. W. Beck and J. V. Jeffery, “Genres of high-stakes writing assessments and the construct of writing

competence,” Assessing Writ, vol. 12, no. 2, pp. 60–79, 2007.[42] M. J. Luzón, “Genre analysis in technical communication,” IEEE Trans. Prof. Commun., vol. 48, no. 3, pp.

285–295, Sep. 2005.[43] J. Johnson-Eilola and S. A. Selber, “The changing shapes of writing: Rhetoric, new media, and composition,”

in Going Wireless: A Critical Exploration of Wireless and Mobile Technologies for Composition Teachers, A. C.Kimme Hea, Ed. Cresskill, NJ: Hampton Press, 2009, pp. 15–34.

[44] B. Thatcher, “Understanding digital literacy across cultures,” in Digital Literacy for Technical Communication:21st Century Theory and Practice, R. Spikla, Ed. New York: Routledge, 2010, pp. 169–198.

[45] C. Spinuzzi, “The genie’s out of the bottle: Leveraging mobile and wireless technologies in qualitative research,”in Going Wireless: A Critical Exploration of Wireless and Mobile Technologies for Composition Teachers, A. C.Kimme Hea, Ed. Cresskill, NJ: Hampton Press, 2009, pp. 255–273.

[46] C. Spinuzzi, Tracing Genres through Organizations: A Sociocultural Approach to Information Design. Cambridge,MA: MIT Press, 2003.

[47] J. D. Bolter and R. Grusin, Remediation: Understanding New Media. Cambridge, MA: MIT Press, 2000.[48] W. Ong, Orality and Literacy: The Technolgizing of the Word. London, UK: Methuen, 1982.[49] J. Murray, Non-Discursive Rhetoric: Image and Affect in Multimodal Composition. Albany, NY: State Univ.

New York Press, 1999.[50] J. D. Bolter and D. Gromala, Windows and Mirrors: Interaction Design, Digital Art, and the Myth of

Transparency. Cambridge, MA: MIT Press, 2003.[51] M. R. Neal, Writing Assessment and the Revolution in Digital Technologies. New York: Teachers College

Press, 2011.[52] J. Rice, The Rhetoric of Cool: Composition Studies and the New Media. Carbondale, IL: Southern Illinois

Univ. Press, 2007.[53] S. K. Langer, Philosophy in a New Key: A Study in the Symbolism of Reason, Rite, and Art, 3rd ed. Cambridge,

MA: Harvard Univ. Press, 1957.[54] L. J. Cronbach and P. E. Mehl, “Construct validity in psychological tests,” Psychol. Bull., vol. 52, no. 4,

pp. 281–302, 1955.[55] M. T. Kane, “Validation,” in Educational Measurement, R. L. Brennan, Ed. 4th ed. Westport, CT: American

Council on Education/Praeger, 2006, pp. 17–64.[56] R. J. Mislevy, R. G. Almond, and J. F. Lukas. A brief introduction to evidence-centered design (Educational

Testing Service RR-03-16). [Online]. Available: http://www.ets.org/Media/Research/pdf/RR-03-16.pdf[57] R. J. Mislevy, “Validity by Design,” Educ. Res., vol. 36, no. 8, pp. 463–469, 2007.[58] E. G. Hansen. Evidence-Centered Design for Learning (RM-11-02). Educational Testing Service. [Online].

Available: http://www.ets.org/research/policy_research_reports/rm-11-02[59] L. Ruth and S. Murphy, Designing Writing Tasks for the Assessment of Writing. Norwood, NJ: Ablex, 1988.[60] C. M. Barnum, Usability Testing Essentials. San Mateo, CA: Morgan Kauffman, 2011.[61] B. Baker, “Playing with the stakes: A consideration of an aspect of the social context of a gatekeeping writing

assessment,” Assessing Writing, vol. 15, no. 3, pp. 133–153, 2010.[62] M. Limayem, S. G. Hirt, and W. W. Chin, “Intention does not always matter: The contingent role of Habit on IT

usage behavior,” presented at the 9th Eur. Conf. Inf. Syst., Bled, Slovenia, Jun. 27–29, 2001.[63] E. A. Gomez, D. Wu, and K. Passerini, “Computer-supported team-based learning: The impact of motivation,

enjoyment, and team contributions on learning outcomes,” Comput. Educ., vol. 55, no. 1, pp. 378–390, 2010.[64] S. Messick, “Validity,” in Educational Measurement, R. L. Linn, Ed. 4th ed. Washington, DC: Amer. Council

Educ. Nat. Council Meas. Educ., 1989, p. 13-1.3.[65] D. Borbsoom, Measuring the Mind: Conceptual Issues in Contemporary Psychometrics. Cambridge, UK:

Cambridge Univ. Press, 2005.[66] P. Deane. Writing Assessment and Cognition (ETS RR-11-14). [Online]. Available: http://www.ets.org/

research/policy_research_reports/rr-11-14[67] E. Wardle and K. Rozen, “Addressing the complexity of writing development: Toward an ecological model of

assessment,” Assessing Writing, vol. 17, no. 2, pp. 106–119.


[68] S. Smith and A. Young, “Adapting to change: Becoming a learning organization as a relief and developmentagency,” IEEE Trans. Prof. Commun., vol. 52, no. 4, pp. 329–345, Dec. 2009.

[69] A. Pacey, The Culture of Technology. Cambridge, MA: MIT Press, 1983.[70] E. A. Gomez, K. Passerini, and K. Hare, “Public health crisis management: Community level roles and

communication options,” presented at the Inf. Syst. Crisis Response Manage. (ISCRAM), Newark, NJ, May 2006.[71] Institute of Medicine, The Future of the Public’s Health in the 21st Century. Washington, DC: The National

Academies Press, 2003.

Elizabeth Avery Gomez is an information communicationtechnology consultant and researcher focusing on interactivecommunication techniques adapted for today’s mobiletechnologies. She specializes in crisis management, focusing onways to adapt communication protocols in support of today’smobile technologies for improved response readiness.

Norbert Elliot is Professor of English at New Jersey Institute ofTechnology, Newark, NJ.