Steven R. Chan MD MBA, John Torous MD, Ladson Hinton MD, Peter Yellowlees MD MBBS

A Framework for Evaluating

Mobile phones are ubiquitous in society and amongst psychiatric patients. Their versatility as a platform can extend mental health services, but the efficacy and reliability of publicly-available applications has yet to be demonstrated. Numerous articles have noted the need for rigorous evaluation of the efficacy and clinical utility of smartphone applications, which are largely unregulated.

Mental Health Smartphone Apps

Clinicians remain concerned about both efficacy and privacy. While there is great potential for mobile applications for psychiatric assessment and interventions, there is limited data on efficacy), scarce research on application security, and little empirical data on outcomes. In essence, little is known about how these applications impact clinical care. However, we also believe another critical factor limits the potential of mobile applications: a lack of standardized evaluation and rating tools that allow clinicians and patients to identify high quality and safe mobile applications. Professional organizations — including the American Psychiatric Association, the American Psychology Association, National Alliance on Mental Illness, and the Substance Abuse and Mental Health Services Administration — do not provide guidelines that patients or clinicians may use to evaluate mobile applications. Consequently, an unregulated free market exists in which many applications are being developed that are of uncertain quality and efficacy. Research on mobile applications for mental health — for instance, smoking cessation — has thus far demonstrated that smartphone applications rarely conform to established guidelines or integrate evidence-based practices. The most thorough meta-analysis of mental health applications to date found only a total of five applications meeting their inclusion criteria, underscoring the lack of thorough evaluation and study of the vast majority of mental health applications on the marketplaces.

Criteria from engineering and informatics disciplines can assist in creating guidelines. Heuristics governing software usability were adapted for ambulance emergency medical services. The Telehealth Usability Questionnaire adopted similar methods in an assessment of a telepsychiatric system for adult autism assessment. The Healthcare Information and Management Systems Society (HIMSS) created checklists for mobile technologies as part of their mHIMSS initiative. The American Telemedicine Association also include similar criteria in the Lexicon of Assessment and Outcome Measures for Telemental Health.

Patients and providers can use the following three dimensions of evaluation criteria for mental health smartphone applications:

Usefulness dimension➔ Validity and Accuracy — Does the application work as

advertised?➔ Reliability — Will the application consistently function

from session to session?➔ Effectiveness — Is the application clinically effective —

with demonstrated improved outcomes — for the target population, disease, or disability?

➔ Time and number of sessions — What time is required for the user to derive some benefit from the application?

Usability dimension➔ Satisfaction and Reward — Is the application

pleasurable and enjoyable to use, or does it discourage repeat use?

➔ Usability — Can the user easily — or with minimal training — use and understand the application?

OBJECTIVE / BACKGROUND

DEFICITS OF APPS

A POTENTIAL SOLUTION

PROPOSED FRAMEWORK

70%

of patients owned a smartphone

POSTER #

➔ Disability accessibility — Is the application usable by those with disabilities (e.g. incorporates screen readers for blind users, closed captions for the hard-of-hearing and deaf communities)?

➔ Cultural accessibility — Does the application work effectively with the user's culture (as defined by factors such as ethnicity and language)?

➔ Socioeconomic and generational accessibility — Does the application take into account socioeconomic status? And, the user's age, with potential implications for the user's digital health literacy?

Integration & Infrastructure dimension● Security — Is the application's data encrypted on the

device and/or in transmission? Is it anonymized or does it contain personal health information? If so, what does it do?

● Workflow integration — Does the application work within its user's workflow?

● Data integration — Does the application share data with other applications, networks, and medical record systems?

● Safety — Does the application take into account patient safety, such as suicidality or homicidality?

Second, smartphone applications can target one or more of the following stages in a provider’s workflow:

education & training referencehistory data input & output physical data input & output diagnosis treatment and intervention patient-provider communication

1

2

3