It’s already happening: Use of translation

apps and websites in health care settings

Results of a study across five NSW Local Health Districts

Ben Harris-Roxas,1,2 Lisa Woodland,3,1 Joanne Corcoran,3 Jane Lloyd,1,4

Mark Harris,1 Rachael Kearns,1,2 Iqbal Hasan1

1. Centre for Primary Health Care and Equity, UNSW Sydney, Sydney

2. South Eastern Sydney Research Collaboration Hub (SEaRCH), Primary, Integrated

and Community Health, South Eastern Sydney Local Health District , Sydney

3. Priority Populations Unit, Primary, Integrated and Community Health, South Eastern

Sydney Local Health District, Sydney

4. Health Equity Research Development Unit, Sydney Local Health District, Sydney.

The issue

Use of translation apps and

websites in health settings

appears to be increasing.

Machine translation may be

inaccurate for health information,

and is markedly less accurate for

some languages.

TAB B

Recurrent neural networks

• Used by almost all translation apps and

websites

• A piece of software that functions as a

predictive model

• They need big training datasets

Image: Sabine Hossenfelder, Frankfurt Institute for Advanced Studies

Recurrent neural networks

• Neural networks typically have hundreds

or thousands of neurons

• The human brain has 100 billion (but

RNNs are faster)

• The hidden layers may be hard for us to

understand and interpret – and may

change over time

• They optimise for a defined output

Image: Sabine Hossenfelder, Frankfurt Institute for Advanced Studies

An example

“You need to drink some fluids

before your colopyopause that

will open your eyes and

remove from stool so that we

can see your face face.”

Potential uses

During clinical interactions

Written resources

General health information

Website information

It is unclear who’s initiating use and

in what circumstances

The context Most research on the development and

accuracy of machine translation (Birch et al 2016,

Koponen et al 2016, Patil and Davies 2014)

Some on types of apps (Panayiotou et al 2019)

Less on use within health care settings (Anazawa et al 2013, Michael et al 2013)

2017 survey of 698 staff in three NSW

Local Health Districts/Specialty Networks

found 18% had used a translation app

Updated NSW Health Policy Directive in

December 2017 (NSW Health 2017)

Anazawa, R., et al. (2013). "Evaluation of online machine translation by nursing users." CIN: Computers, informatics, Nursing 31(8): 382-387.

Birch, A., et al. (2016). "HUME: Human UCCA-based evaluation of machine translation." arXiv preprint arXiv:1607.00030.

Koponen, M. (2016). "Is machine translation post-editing worth the effort? a survey of research into post-editing and effort." The Journal of Specialised

Translation 25: 131-148.

Michael, J., et al. (2013). "Development of a Translation Standard to support the improvement of health literacy and provide consistent high-quality information."

Australian Health Review 37(4): 547-551.

NSW Ministry of Health. Interpreters – Standard Procedures for Working with Health Care Interpreters [Policy Directive PD2017_044]. 2017.

Panayiotou, A., Gardner, A., Williams, S., Zucchi, E., Mascitti-Meuter, M., Goh, A.M.Y., You, E., Chong, T.W., Logiudice, D., Lin, X., Haralambous, B., Batchelor,

F., 2019. Language Translation Apps in Health Care Settings: Expert Opinion. JMIR mHealth and uHealth 7, e11316.

Patil, S. and P. Davies (2014). "Use of Google Translate in medical communication: evaluation of accuracy." BMJ 349: g7392.

Research question

What is the nature and extent of

translation app and website use

in state funded health care

services in NSW?

Staff survey and semi-structured

qualitative interviews

Survey sample 1,558 respondents

5 Local Health Districts

2.0% response rate

80.3% female, 18.2% male

70.7% clinical staff, 29.3% non-clinical

Interview sample 24 participants

5 Local Health Districts

Range of health professions, managers and

senior managers/policy roles

Survey findings

App users33.6% had used a translation app or

website in a clinical encounter (n=516)

Of these:

75.4% had used a translation app within

the past 12 months (57.4% in past 3

months)

Clinicians initiated the use of a

translation app in two-thirds of their most

recent experience (66.8%)

72.8% has most recently used an app

between 8:30am and 5pm

42.6% used an app after a request for a

professional interpreter had been made.

Survey findings

App users

Younger

44.2% aged under 40 had used translation

apps, compared to 28.3% aged 40+. χ(1)=40.181, p<0.001

Male

43.0% of males had used translation apps,

compared to 28.3% of people who identified

as female.

N.B. sample 80% female, χ(1)=10.676, p=0.001

Less experienced

39.6% who had worked for NSW Health for

less 10 years had used translation apps,

compared to 30.1% who had worked for

10+ years. χ(1)=14.759, p<0.001

Clinical staff

41.8% of clinical staff had used translation

apps, compared to 16.4% of non-clinical

staff. χ(1)=87.062, p<0.001

Survey findings

App users by profession

63.50%

41.9%

29.8%

17.8%

0%

10%

20%

30%

40%

50%

60%

70%

Medical Nursing/midwifery Allied health Administration

Proportion within profession using apps (%)

χ(10)=151.935, p<0.001

Survey findings

App users by setting

50.5%

45.0%42.3%

39.3%

30.7%

11.9%

0%

10%

20%

30%

40%

50%

60%

Proportion of respodents within setting (%) χ(10)=89.588, p<0.001

Survey findings

Perceptions of app use

93.4% rated the translation app as very

useful or useful.

57.8% rated the risk of misinterpretation as

low or none.

33.5% rated the translation as accurate or

very accurate.

Interview findings

Risks being weighed up: Inaccuracy and time

“But the patient expressed himself in a way that I felt that I

almost certainly understood. It was congruent with what is

his hand gestures and his behaviour was I thought I

understood the response. But it was a very simple

communication. I think anything that was about symptoms

would be very difficult to, to rely on a translation app.”

Interview findings

Risks being weighed up: Inaccuracy and time

“We weren't able to book a face to face Thai interpreter on

time, and our patient was already on our bed so we weren't

able to physically pull the phone over to her dial the phone

interpreter and wait for them because there is also a chance

that we might not be able to get a phone Thai interpreter

either. Plus the use was only for some basic questions, so

we thought it would be okay.”

Interview findings

Risks being weighed up: Inaccuracy and time

“I think definitely “time-constraints” is a factor. We can't call

interpreter service on the phone every single day to ask a

patient every single day, and some patients just require a

little bit more communication than others, and we can't rely

on interpreters services every time when a patient has a

question, and they want it to be answered right there right

then, or if the patients really busy and they need to leave

very soon but they have questions in their mind. We can't

provide them with an interpreter every single time. Basically I

think it’s down to the best thing that we can do.”

Interview findings

Risks being weighed up: Inaccuracy and time

“Different cultures explain things differently. There's no direct

translation you have to understand a language, you can't

just use a word. So you can't type in a word and expect it,

guaranteed accurate.”

Interview findings

Risks being weighed up: Inaccuracy and time

“Anything that's emotive or involves more risk, I think, it's

imperative that you have accurate interpreting. But I think in

health generally it's really important to be accurate in your

language translations. I don't think non-human ways of doing

it are good enough at the present.”

Discussion Use appears to be widespread in state

funded health care services in NSW.

In most cases clinicians are initiating use.

Use may be growing. In the 2017 survey

18% reported app use. For the 301

respondents from a similar (but not exactly

the same) sample area in this survey,

38.2% reported app use.

Contradiction about perception that app

translation is not accurate, but that the

potential risks of this are low.

People are making trade-offs.

Limitations Low response rate.

Potential for self-censorship, actual rates of

use may be higher.

What remains unknown?

Is there variability in adoption in other

settings?

What are health consumers, family and

carers’ experiences and perceptions?

Is a more integrated research approach

may be required?

(Intersection of data science,

communication and linguistics, psychology,

and health and social service delivery)

The difficulty in developing

feasible and realistic policy

responses and guidance

Privacy and data governance

O’Leary, D. E. (2008). Gartner’s hype cycle and information system research issues. International Journal of Accounting Information Systems,

9(4), 240–252. https://doi.org/10.1016/j.accinf.2008.09.001

Where are we up to?

Where are we up to?

May, C. R., Cummings, A., Girling, M., Bracher, M., Mair, F. S., May, C. M., … Finch, T. (2018). Using Normalization Process Theory in

feasibility studies and process evaluations of complex healthcare interventions: A systematic review. Implementation Science, 13(1), 80

Deep learning

(or poor metaphors?)

“At the same time, deep learning has no good way to

incorporate background knowledge. A system can learn to

predict that the words wallet and safe place occur in

similar kinds of sentences ("He put his money in the

wallet," "He put his money in a safe place"), but it has no

way to relate that to the fact that people like to protect

their possessions.”

Source: Marcus G, Davis E. (2019) Rebooting AI: Building Artificial Intelligence We Can Trust,

Pantheon: New York.

Image: Marketa

“Super-linguists” will be needed

to trim the linguistic engines

and, eventually, override the

recommendations of machines.

For a long time still, complex

semantics and the interpretation

of text passages will remain the

art of human involvement in

processing the input and output.

Technology is not set to become

autonomous in the medium

term. It can serve people very

well if applied with boldness and

creativity, as well as with

responsibility.”

Source: Svoboda, T. (2017). No linguistic borders

ahead? Looking beyond the knocked-down

language barrier. TranscUlturAl: A Journal of

Translation and Cultural Studies, 9(2), 86–108.

https://doi.org/10.21992/T93Q0FImage: Robert Couse-Baker

The future?

Moving beyond the

transmission model

Health literacy

The importance of community and

culture

Genuine person-centred

approaches

Image: Olaf Eichler

What’s at stake?

Acknowledgements

Funders

Priority Populations Unit, South Eastern Sydney Local

Health District; NSW Ministry of Health

Site investigators

Sue Buckman, Vesna Dragoje, Eva Melham, Bradley

Warner, Ashley Young

Collaborators

SESLHD; Health and Social Policy Branch, NSW

Ministry of Health; WSLHD Translation Unit; SLHD,

SHCIS; SWSLHD; Refugee Health Service, HNELHD,

Multicultural Health Service, NSW Multicultural Health

Communications Service

Thank you

Ben Harris-Roxas

Email b.harris-roxas@unsw.edu.au

These slides are available at

https://www.slideshare.net/benharrisroxas