ORIGINAL ARTICLE – HEALTHCARE POLICY AND OUTCOMES

Developing and Testing TEAM (Team Evaluation and AssessmentMeasure), a Self-assessment Tool to Improve CancerMultidisciplinary Teamwork

C. Taylor, MA1, K. Brown, PhD2, B. Lamb, MRCS2,3, J. Harris, MSc1, N. Sevdalis, PhD2, and

J. S. A. Green, FRCS3,4

1Florence Nightingale School of Nursing and Midwifery, King’s College London, London, England; 2Department

of Surgery and Cancer, Imperial College London, London, England; 3Department of Urology, Whipps Cross University

Hospital, London, England; 4Faculty of Health and Social Care, London South Bank University, London, England

ABSTRACT

Background. Cancer multidisciplinary teams (MDTs) are

well established worldwide and are an expensive resource

yet no standardised tools exist to measure performance. We

aimed to develop and test an MDT self-assessment tool

underpinned by literature review and consensus from over

2000 UK MDT members about the ‘‘characteristics of an

effective MDT.’’

Methods. Questionnaire items relating to all characteristics

of MDTs (particularly Leadership and Chairing; Team-

working and Culture; Patient-centred care; Clinical

decision-making process; and Organisation and adminis-

tration during meetings) were developed by an expert panel.

Acceptability, feasibility and psychometric properties were

tested by online completion of the questionnaire by 23 MDTs

from 4 UK NHS Trusts followed by interviews with 74 team

members including members from all teams and nonre-

sponders. 10 of the MDTs also completed questionnaires that

directly translated each characteristic to an item (for the five

domains above) to test content validity.

Results. A total of 47 items were created, each rated for

agreement on a 5-point scale. A total of 329 (52 %) of 637

team members completed the questionnaire, including

representation from medical, nursing and clerical MDT

members. Responses correlated well with domain-specific

questionnaires (r [ 0.67, p = 0.01), most domain-scales

had acceptable internal consistency (Cronbach alpha

[ 0.60), and good item discrimination (majority of items

r \ 0.20). Team members were positive about its value.

Conclusions. Self-assessment of team performance using

this tool may support MDT development.

The increasing complexity of treatment and manage-

ment decisions for cancer patients has led to the need for

the relevant nursing, surgical, medical and diagnostic

experts to work closely together in order to optimise patient

care. As a consequence, multidisciplinary teams (MDTs)

are firmly established at the core of cancer care in the UK

and in many other countries worldwide. In the UK it is

mandatory for all new patients to be discussed in weekly

MDT meetings (akin to multidisciplinary case conferences

or tumour boards) where all relevant clinical and patient-

based information should be shared and discussed and

treatment recommendations agreed.

The concept of the MDT varies among countries though

many countries have a forum for ensuring that relevant

expertise is included when making treatment recommenda-

tions.1 In England MDTs are organised within 28 cancer

networks. Care is provided in local hospitals by a local MDT

(members of which may be based in one or more hospital)

who are able to manage and treat the more common cancers

and less complex presentations of cancer. For rarer cancers

and more complex cases, there exist specialist teams that are

based (or hosted) in cancer centres. Generally there is at least

one cancer centre per cancer network. The membership and

structure of MDTs is tumour-specific and detailed in

Improving Outcomes Guidance (published by NICE).

MDTs are a very expensive resource. There are

approximately 1500 cancer MDTs in England, and based

simply on data about the time taken for radiologists and

� Society of Surgical Oncology 2012

First Received: 27 February 2012;

Published Online: 21 July 2012

C. Taylor, MA

e-mail: cath.taylor@kcl.ac.uk

Ann Surg Oncol (2012) 19:4019–4027

DOI 10.1245/s10434-012-2493-1

pathologists to prepare for MDT meetings, it has been

estimated they cost £100 million a year for attendance and

preparation time.2,3 This investment in resources has to-

date been unmatched by robust evidence of their effec-

tiveness. Indeed we know very little about how well MDTs

are functioning either in or outside the weekly meetings. In

England, MDTs are subject to a mandatory self-assessment

and independent peer review process whereby every MDT

has to provide evidence to demonstrate compliance with

tumour-specific best practice, and may also be reviewed by

an independent panel of health professionals and patient

representatives (http://www.cquins.nhs.uk).4 Data from this

process provide evidence of wide variability in perfor-

mance in relation to the standards assessed—including

standards for the structure of MDTs (e.g., membership and

attendance at meetings) and having relevant protocols in

place for referral and treatment.5

In addition, there is evidence that MDTs also vary in

relation to their team processes—such as leadership,

teamworking and team decision making. Such processes

are not captured via the UK peer review system, but have

shown to be instrumental for optimising decision mak-

ing.6,7 MDT meeting discussions influence the quality of

diagnostic and treatment decisions.8–10 This is likely to

require inclusive discussion across a range of occupational

groups.11 Poor teamworking in MDMs may lead to non-

inclusive discussions resulting in information bias, with

discussions centred on the tumour rather than the per-

son.12,13 Failure of the team to consider all relevant

information may lead to poor decisions, or to decisions

which are not implemented as they are unaccept-

able to patients or clinically inappropriate.14–17 Non-

implementation of MDT meeting recommendations can

have both clinical and financial consequences if further

discussion is required and treatment is delayed.

In order to optimise team performance and team deci-

sion making in cancer MDTs it is necessary first to develop

valid performance measurement tools to enable teams to

identify where they are performing well and where they

could improve.17 Within health care, an expanding body of

evidence shows that team performance relates to patient

safety as well as other parameters of quality in patient

care.18,19 Team assessment tools and relevant training

modules have been developed and systematic reviews

demonstrating their efficacy are now available, though

little of this work has involved cancer MDTs.20–23

Assessment tools have the added advantage that they can

be both diagnostic and interventional if used to provide

structured feedback. This technique has yet to be evaluated

systematically for cancer MDTs though has been shown to

improve performance in other health care teams.24

A valid self-report instrument that enables an MDT to

comprehensively self-assess the core functions of their team

and team meetings is desired by team members.25 In 2010, the

National Cancer Action Team (NCAT) published ‘‘The

Characteristics of an Effective MDT,’’ comprising recom-

mendations for effective MDT-working based upon

consensus from over 2000 MDT members.25,26 We aimed to

develop and test the acceptability, feasibility and psychometric

properties of a team assessment questionnaire, underpinned by

the ‘‘Characteristics of an effective MDT’’ and intended as a

stimulus to team self-assessment and improvement.

METHODS

Development of TEAM

On the basis of preliminary work undertaken with over

60 MDTs (over 300 team members) it was determined that

MDT assessment should include a 360-degree question-

naire completed by all individual team members to act as a

discussion stimulus (unpublished data). Intended to be

developmental rather than judgemental, it was agreed that

the resulting data would be shared with team members only

(in the first instance) in an anonymised format.

Individual questionnaire items were created by an expert

panel that included members with clinical expertise (J.G.,

B.L.) and academic expertise (C.T., K.B., N.S.) in both

MDT working and questionnaire development. The ques-

tionnaire content was informed by the research evidence

regarding influences on effectiveness in MDT; the recom-

mendations within the ‘‘Characteristics of an Effective

MDT’’ report; and results from the national survey com-

pleted by over 2000 MDT members in the UK.3,6,25–27

The Characteristics report organises recommendations

for optimal MDT-working under 5 domains: (i) the team;

(ii) infrastructure for meetings; (iii) meeting organisation

and administration; (iv) patient-centred clinical decision-

making; and (v) clinical governance.26 These 5 domains

are further separated into 17 subdomains (Table 1). We

used the 5 domains/17 subdomains structure for our survey.

Forty-two questionnaire items were created. Items cov-

ered all 17 subdomains of teamworking with particular

emphasis on Leadership and Chairing; Teamworking and

Culture; Patient-centred care; Clinical decision-making

process; and Organisation and administration during

meetings. This was due to (i) evidence of their importance

to optimal patient care; (ii) not being assessed through the

National Peer Review Programme; and (iii) confirmation in

preliminary work involving over 60 MDTs that team

assessment should prioritise these domains (unpublished

data). In addition, the expert panel developed five items

aimed at measuring team members’ general perceptions of

the effectiveness of their team including one ‘‘global’’

item. All 47 items were designed to be rated on 5-point

4020 C. Taylor et al.

agreement scales (anchored at 1 = completely disagree,

5 = completely agree). A ‘‘don’t know’’ option was also

included. Finally, two free-text questions were also inclu-

ded asking respondents to state at least three areas where

they felt their team worked well and three areas they could

improve upon. The publication rights to TEAM are owned

by the NCAT. Sample items for each domain are provided

in Table 1.

Testing Acceptability, Feasibility, and Psychometric

Properties

Sample 23 MDTs from 4 NHS Trusts were recruited to

test the questionnaire. A Trust is the provider of health care

services and in most cases equates to a hospital site. Teams

were recruited via their Trust cancer services manager, who

in turn had approached NCAT or the research team to

express an interest to be involved. Each Trust cancer

services manager was asked to recruit five MDTs willing to

participate (one Trust provided seven MDTs, and another

provided six).

Procedure The survey was administered online via

SurveyMonkey (http://www.surveymonkey.com/). All

team members were sent a link to an online questionnaire

via e-mail using the automated invitation system provided

by SurveyMonkey and given approximately 3 weeks to

respond. Non-responders received a maximum of two

weekly e-mail reminders. The second reminder also

included a Microsoft Word version of the survey in case

non-response was due to technological problems.

Content of Online Survey All team members completed

background questions (professional group and whether

they were the MDT-lead or chair for the team) and the

TEAM questionnaire. In order to test the content validity of

the questionnaire, team members from 10 of the MDTs

(from two NHS Trusts) also completed a second

questionnaire (further details follow).

TABLE 1 Sample TEAM items according to domains/subdomains of teamworking

Domain of teamworkinga Subdomain (no. of TEAM items)a Sample TEAM items

The team Membership (1)

Attendance (2)

Leadership & Chairing (5)

Teamworking & Culture (9)

Personal development &

Training (2)

Core team members or their deputies are always present at our

MDT Meetings

Leadership could be improved in our Team

I feel part of a true team—we are more than a group of

individuals who come together for the weekly MDT meeting

If training needs are identified for Multidisciplinary Team

members, action is taken to address those needs

Infrastructure for meetings Physical environment of meeting

venue (1)

Technology & equipment (1)

My Trust provides standard of equipment for interhospital

compatibility, real-time viewing of imaging and pathology,

and documentation of recommendations

Organisation and administration

for meetings

Scheduling of MDT meetings (2)

Preparation prior to MDT

meetings (2)

Organisation/administration

during MDT meetings (4)

Post-MDT meeting coordination

of service (1)

I am able to attend our MDT Meeting regularly, and this is

reflected in my job plan

Locally agreed minimum data sets of information are prepared

and cases are prioritised in a logical order at our MDT

Meetings

Clinical information is presented to a high standard at our

MDT Meetings

Patient-centred clinical decision

making

Who to discuss? (1)

Patient-centred care (4)

Clinical decision-making

process (4)

Someone the patient has met always attends the MDT Meeting

to discuss their case

I would like my Team to look after me if I were a cancer

patient

Team governance Organisational support (1)

Data collection, analysis & audit

of outcomes (1)

Clinical governance (1)

Our Team collects information and reviews it in order to

continually improve on equality issues and clinical

outcomes, and benchmark against best practice

We compare and audit our Team’s recommendations against

the actual treatment received and any serious complications

that may occur

From ‘‘The characteristics of an effective MDT’’ (NCAT, 2010)26

TEAM to Self-assess MDTs 4021

Psychometric Properties

Content Validity Questionnaires were developed for each

of the five subdomains prioritised for inclusion: leadership

and chairing (29 items); teamworking and culture (11);

patient-centred care (14); clinical decision-making process

(24); organisation and administration for meetings (26).28,29

These were created by translating each recommendation into

a statement rated for agreement using the same 5-point scale

as TEAM. In order to minimise response burden team

members (from the 10 MDTs participating in this validation

exercise) received a questionnaire containing a maximum of

three out of the five subdomains. Average ratings for each

subdomain were calculated by summing the ratings given to

individual items within the scale and dividing by the number

of items. The correlation of average ratings between TEAM

responses and the domain-specific questionnaires was then

measured by calculating Pearson’s correlation coefficients.

Internal Consistency Internal consistency was examined

for the subdomains within the 47-item questionnaire and for

each of the domain-specific questionnaires by calculating

Cronbach’s alphas. Typically acceptable alphas ought to

reach 0.70 or higher; lower alpha values (i.e., between 0.60

and 0.70) can be accepted for short scales and for preliminary

research purposes.30

Item Discrimination In order to assess for potential

(statistical) overlap between items, the correlation between

each individual item within the 47-item questionnaire was

measured by calculating Pearson’s correlation coefficient.

Acceptability and Feasibility

The acceptability and feasibility of TEAM was assessed

by conducting telephone interviews with a purposive

sample of approximately four team members from each

team after they had completed TEAM and discussed their

team responses. Individual team members were sampled to

represent different professional groups; those who com-

pleted/did not complete the online survey; and included

MDT leads/chairs.

Interviews were semi-structured following a topic-guide

aimed at determining the acceptability of the length and

content of the questionnaire and feasibility of online com-

pletion. They were digitally recorded, and interview data

were coded directly from recordings by one of four

researchers (all trained in qualitative analytic methods).

Codes were applied as appropriate and illustrative quotations

taken from each interview in relation to each relevant code.

A subset of recordings (n = 8) were double-coded by a

researcher with substantial experience of the project who

confirmed the reliability of coding (88 % agreement).

Ethical Review

The protocol for this work was reviewed by UK

National Research Ethics Service (NRES) and confirmed to

be classified as service development and not eligible for

review.

RESULTS

Participants

In total the survey invitation was e-mailed to 637 team

members across the 23 teams, including core and extended

team members. A total of 329 team members (52 %)

responded to the survey (Table 2). Forty-eight percent of

team members did not even open the survey link. Of those

who opened the survey link, 77 % answered the whole

questionnaire and 90 % answered at least 40 items. In all

teams there was good representation of the main profes-

sional groups amongst those who responded (medical,

nursing and administrative members).

The response rates were similar across the 4 Trusts

(range 48–59 %). The range for individual teams was

26–83 %, and both the highest and the lowest response

rates came from the same Trust (Trust D). There was a

wide variation in response rates within and between tumour

types (colorectal 35–63 %; gynae 60–83 %; head and neck

26–58 %; urology 45–52 %; lung 53–63 %). By profes-

sional group, the highest response rate was from clinical

nurse specialists (59 %). Other types of nurses, managers

and allied health professionals (AHPs) were most likely to

not even open the survey link, and Managers and AHPs

were the most likely to stop completing the survey after the

background items had been completed.

Acceptability, feasibility and psychometric properties

of the questionnaire

Content Validity Responses to the 47-item questionnaire

were strongly and significantly correlated with responses to

the domain-specific questionnaires for all 5 domains that

were tested (r [ 0.67, p \ 0.01) (Table 3).

Internal Consistency The scales for two domains

(Leadership and Teamwork and Culture) had acceptable

alpha ratings (Cronbach alpha [ 0.70) with a further five

domains achieving alpha ratings C0.60 (Table 4).

Attendance, Scheduling of MDT Meetings, and

Preparation for MDT meetings, each having only two

items, had alpha ratings between 0.52 and 0.59. The

domain-specific questionnaires all had very high internal

consistency ([0.89).

4022 C. Taylor et al.

Item Discrimination There was little correlation between

responses to individual items in the 47-item questionnaire.

Only 4 items had a moderate correlation of r = 0.60–0.65:

‘‘I feel part of a first class team,’’ ‘‘There is mutual respect,

trust and healthy debate between our Multidisciplinary

Team members,’’ ‘‘I would like my Team to look after me if

I were a cancer patient,’’ and ‘‘I feel part of a true team—

we are more than a group of individuals who come

together for the weekly MDT meeting.’’ The majority of

items had weak correlations (r \ 0.20), indicating that

items are each measuring discrete aspects of teamworking

(analysis available on request).

Acceptability and Feasibility of the Questionnaire Completion

of TEAM (based on SurveyMonkey time logs) took on

average 17 minutes (standard deviation 11 minutes).

In total, 74 telephone interviews were conducted with

a purposively selected range of team members. All

professional groups were well represented. This included

interviews with 15 team members that had not responded

to the online survey. The reasons given by these team

members for not completing the survey were explained

by at least one of three themes: (1) being on annual

leave (the testing phase coincided with two consecutive

UK bank holiday weekends); (2) not having time to

dedicate to it: ‘‘Just [not enough] time and workload,

had been on holiday previously—it was not that I didn’t

want to fill it in’’; and (3) having technical problems

with the online system: ‘‘I couldn’t get into it—clicked

on the link and it didn’t work and then forgot to go back

to it.’’ One team member stated that the bank holidays

had led to a backlog of work as part of the reason for

him not participating and then also added that he ‘‘knew

a lot of other people completed it and felt as though not

everyone needed to complete it.’’

The majority of team members were positive about the

questionnaire (51 of 74, 69 %, of team members inter-

viewed). This included positive comments about the

TABLE 2 Response rate to

online survey according to Trust

and team

a A total of 253 (83 %) of 306

participants answered all 47

items; 296 (97 %) of 306

participants answered 40?

items

Trust/Team No. members

contacted

Answered at least

some of TEAMaAnswered demographic

items only

Did not open

survey link

Trust A 118 62 (53 %) 8 (7 %) 48 (40 %)

Breast 14 8 (57 %) 2 (14 %) 4 (29 %)

Colorectal 17 6 (35 %) 1 (6 %) 10 (59 %)

Gynae 20 12 (60 %) 2 (10 %) 6 (30 %)

Head and neck 24 14 (58 %) 2 (8 %) 8 (33 %)

Urology 15 7 (47 %) 1 (7 %) 7 (47 %)

Lung 16 10 (63 %) 0 (0 %) 6 (38 %)

Lymp/Haem 12 5 (42 %) 0 (0 %) 7 (58 %)

Trust B 117 59 (50 %) 2 (2 %) 56 (48 %)

Colorectal 28 15 (54 %) 0 (0 %) 13 (46 %)

Head and neck 31 11 (36 %) 1 (3 %) 19 (61 %)

Lung 15 8 (53 %) 0 (0 %) 7 (47 %)

Skin 10 8 (80 %) 0 (0 %) 2 (20 %)

Urology 33 17 (52 %) 1 (3 %) 15 (46 %)

Trust C 179 86 (48 %) 4 (2 %) 89 (50 %)

Head and neck 52 24 (46 %) 0 (0 %) 28 (54 %)

HPB 36 19 (53 %) 1 (3 %) 16 (44 %)

Lung 29 16 (55 %) 0 (0 %) 13 (45 %)

Sarcoma 33 14 (42 %) 1 (3 %) 18 (55 %)

Urology 29 13 (45 %) 2 (7 %) 14 (48 %)

Trust D 223 99 (44 %) 9 (4 %) 115 (52 %)

Gynae 29 24 (83 %) 0 (0 %) 5 (17 %)

Head and neck 47 12 (26 %) 2 (4 %) 33 (70 %)

Colorectal (a) 36 13 (36 %) 1 (3 %) 22 (61 %)

Upper GI 35 12 (34 %) 3 (9 %) 20 (57 %)

Urology 57 26 (46 %) 3 (5 %) 28 (49 %)

Colorectal (b) 19 12 (63 %) 0 (0 %) 7 (37 %)

Total 637 306 (48 %) 23 (4 %) 308 (48 %)

TEAM to Self-assess MDTs 4023

TA

BL

E3

Co

nte

nt

val

idit

y:

corr

elat

ion

bet

wee

nth

era

tin

gs

for

team

per

form

ance

bet

wee

nT

EA

Man

dd

om

ain

-sp

ecifi

cq

ues

tio

nn

aire

s

Do

mai

n-s

pec

ific

qu

esti

on

nai

reT

eam

Gen

eral

feel

ing

s

Lea

der

ship

&

chai

rin

g

Cli

nic

al

dec

isio

n-m

akin

g

Pat

ien

t

cen

tred

care

Tea

mw

ork

ing

&

cult

ure

Org

anis

atio

n&

adm

ind

uri

ng

mtg

s

Per

son

ald

evel

op

men

t

and

trai

nin

g

Mem

ber

ship

Pre

par

atio

n

pri

or

tom

tgs

Lea

der

ship

&ch

airi

ng

.54

**

,a.8

0*

*,c

.52

**

,a.4

0*

*.6

4*

*,b

.58

**

,a.5

9*

*,a

.23

.53

**

Lea

der

ship

on

ly.4

9*

*.6

9*

*,b

.53

**

,a.4

0*

*.6

8*

*,b

.47

**

.58

**

,a.2

9.4

0*

*

Ch

airi

ng

on

ly.5

1*

*,a

.81

**

,c.4

5*

*.3

6*

.60

**

,a.5

4*

*,a

.50

**

,a.1

4.5

3*

*,a

Pat

ien

tce

ntr

edca

re.3

2*

.36

*.6

7*

*,b

.67

**

,b.4

0*

*.2

4.0

8.2

2.3

1*

Cli

nic

ald

ecis

ion

-mak

ing

.49

**

.59

**

,a.7

9*

*,c

.68

**

,b.5

7*

*,a

.44

**

.34

*.3

5*

.46

**

Tea

mw

ork

ing

and

cult

ure

.61

**

,b.5

5*

*,a

.63

**

,b.6

4*

*,b

.79

**

,c.7

0*

*,c

.57

**

,a.0

4.5

4*

*,a

Org

anis

atio

nan

dad

min

du

rin

gm

tgs

.61

**

,b.6

2*

*,b

.63

**

,b.6

7*

*,b

.61

**

,b.7

7*

*,c

.68

**

,b.0

6.5

8*

*,a

Do

mai

n-s

pec

ific

qu

esti

on

nai

reT

eam

Sch

edu

lin

gA

tten

dan

ceC

lin

ical

go

ver

nan

ce

En

vir

on

men

tT

ech

no

log

yP

ost

-mtg

acti

on

s

Wh

oto

dis

cuss

Org

anis

atio

nal

sup

po

rt

Dat

aco

llec

tio

n&

aud

it

Lea

der

ship

&ch

airi

ng

.24

.39

**

.41

**

.22

-.1

0.2

1.4

9*

*.0

8.4

9*

*

Lea

der

ship

on

ly.3

5*

.38

**

.35

*.3

0*

-.0

3.1

3.3

5*

.08

.54

**

,a

Ch

airi

ng

on

ly.1

9.3

8*

*.4

0*

*.1

8-

.11

.26

.46

**

.03

.50

**

,a

Pat

ien

tce

ntr

edca

re.3

3*

.46

**

.34

*.2

6.0

7.3

3*

.25

-.1

7.4

6*

*

Cli

nic

ald

ecis

ion

-mak

ing

.38

**

.46

**

.42

**

.34

*.0

5.2

2.4

3*

*-

.02

.40

*

Tea

mw

ork

ing

and

cult

ure

.40

**

.60

**

,b.2

3.4

0*

*.4

1*

*.3

6*

*.3

5*

.16

.29

*

Org

anis

atio

nan

dad

min

du

rin

gm

tgs

.34

**

.68

**

,b.3

4*

*.4

6*

*.5

9*

*,a

.38

**

.34

*.0

5.3

8*

*

n=

39

–6

5.

An

aly

ses

incl

ud

em

emb

ers

wh

oco

mp

lete

dal

lit

ems

wit

hin

the

scal

esco

rrel

ated

.E

ach

do

mai

n-s

pec

ific

qu

esti

on

nai

rew

asse

nt

tom

emb

ers

fro

mfi

ve

MD

Ts

aC

orr

elat

ion

[0

.50

bC

orr

elat

ion

[0

.60

cC

orr

elat

ion

[0

.70

*p\

0.0

5(2

-tai

led

)

**

p\

0.0

1

4024 C. Taylor et al.

content, length and the 360-degree method enabling

everyone to contribute (Table 5). A common theme was

the importance of team engagement to ensure participation

in the process, and also ensuring that the IT works properly

(this was particularly mentioned by some team members

from one Trust who had technical difficulties due to poor

Trust IT systems).

DISCUSSION

We have developed a team performance assessment

questionnaire (TEAM) based upon input from over 300

team members from over 60 MDTs. The questionnaire is

underpinned by recommendations contained within ‘‘The

Characteristics of an Effective MDT,’’ which in turn is

TABLE 4 Internal consistency of scales within TEAM and the domain-specific questionnaires

Subdomain of teamworking (scale) TEAM items Domain-specific items

No. of items Item mean (SD) Alpha No. of items Item mean (SD) Alpha

Attendance 2 4.04 (0.32) 0.59 NA NA NA

Leadership & chairing 5 3.77 (1.08) 0.76a 29 3.80 (0.81) 0.97a

Teamworking & culture 9 3.99 (1.01) 0.81a 11 3.87 (1.00) 0.89a

Personal development & training 2 3.45 (0.17) 0.60b NA NA NA

Scheduling of MDT meetings 2 3.92 (0.45) 0.59 NA NA NA

Preparation prior to MDT meetings 2 3.75 (0.19) 0.52 NA NA NA

Organisation admin during MDT mtgs 4 3.69 (0.67) 0.62b 26 3.44 (1.88) 0.94a

Patient-centred care 4 3.77 (1.13) 0.62b 14 3.98 (0.62) 0.95a

Clinical decision-making process 4 4.17 (0.33) 0.65b 24 3.95 (1.15) 0.95a

General feelings 4 3.74 (0.60) 0.65b NA NA NA

Based on data from 22 teams (n = 225–291 for TEAM items, n = 16–57 for domain-specific items). Single item domains removed (Mem-

bership; Physical environment of meeting venue; Technology & equipment; Postmeeting coordination of services; Who to discuss;

Organisational support; Data collection, analysis & audit; Clinical governance)a Alpha C 0.70b Alpha C 0.60

TABLE 5 Acceptability and feasibility of TEAM

Theme (no. of team members

supporting theme)

Illustrative quotations

Content: good content and coverage

(35)

‘‘It’s thorough and an easy survey to complete. You feel motivated to fill in, as you want to know what

other people in the team think’’

Too long, did not like aspects of

content (6)

‘‘The questions were clear and it wasn’t too onerous’’

‘‘It didn’t take me that long at all, it was fine’’

‘‘The survey provided an opportunity to reflect on the running of our MDT and the actual functionality of

MDT process, how it works, and those things not covered by peer review’’

‘‘I thought some of the questions were a bit unclear’’

‘‘It took longer to fill in than I thought it would’’

Method: 360-degree process is

useful/reassuring (23)

‘‘The 360 degree feedback, overall it was reassuring that there was a lot of consensus that things are going

well, and very reassuring as the lead that it’s not just my opinion’’

‘‘[360 degree] gave overall picture about what people think of it’’

‘‘It gives people an opportunity to judge how they feel the MDT process works in an anonymous way, and

it allows evaluation of the strengths and weaknesses of the team’’

Importance of team engagement

(12)

‘‘Ownership in the process, feeling it has relevance, and a belief that the process will make a difference is

important’’

‘‘It’s important to engage people to participate, there must be someone to drive it, if there is a lack of

realisation within the team of importance of process then people may treat as a tick box exercise. The

lead needs to be able to push to ensure its completed by the team’’

Importance of good IT (19) ‘‘You are never going to get a 100 % response rate . . . but you need good IT and need it to work on the

iPad’’

‘‘The survey was slow changing screens, the internet speed varies in the Trusts so it’s probably down to

that, but it was very slow’’

TEAM to Self-assess MDTs 4025

based on clinical consensus from over 2000 MDT members

nationwide.26

TEAM has been tested with 23 MDTs from four hospital

Trusts and has been found to be acceptable in content and

length, and to have reasonable psychometric properties

including content validity, internal consistency and item

discrimination. It has been designed and tested as part of a

broader team assessment process within which all teams

were able to identify areas for improvement of teamworking

and some changes were implemented immediately.31

To-date TEAM has mostly been tested by local cancer

teams treating common tumours. Adaptation of TEAM

may be necessary to ensure coverage of the characteristics

of effectiveness in teams with different team structures or

processes (e.g., for specialist teams working across multi-

ple sites and using videoconferencing for their weekly

meetings). It may also be necessary to review the items for

scales that did not reach acceptable levels of internal

consistency. Furthermore, whilst we have designed TEAM

on the basis of clinical consensus from over 2000 MDT

members regarding the features of an effective MDT, these

characteristics of effectiveness have yet to be validated

against outcomes—including clinical processes, patient

outcomes and patient experience. Given the complexity of

cancer as a disease but also its care pathway, some of these

measures may be more or less sensitive to correlations with

TEAM. Examining the relationship between TEAM and a

range of clinical processes and outcomes, including patient

experience may help validate the clinical consensus

regarding key ingredients of effectiveness.18

In addition, although we report generally positive views

regarding the content and length of TEAM, on average

only just over half of team members completed it. Team

members were only given a short time (maximum of

3 weeks) to respond and this coincided with UK national

holidays. The majority of those who opened the link

completed the survey (83 % answered whole survey and 97

% answered at least 40 of 47 items). Compared to postal

surveys, online surveys may be more easily ignored or

deleted, less attention-grabbing; and feel less anonymous.32

Interviews with a number of non-responders established

that their non-participation was mostly the result of lack of

time and/or workload pressures at that time rather than any

of these reasons, and they confirmed the representativeness

of their team feedback reports indicating that a full

response rate may not have changed the eventual team

outcomes. The ease of completion of e-surveys may

therefore outweigh any potential downside. Although not

cited as a particular reason for non-response, there were

some technological issues that may have impeded partici-

pation. Development of a format that is compatible with

NHS IT resources and/or enables team members to par-

ticipate using different forms of technology, such as

smartphones, would be necessary for TEAM to have wider

utility and acceptability.

Team members emphasised the importance of both team

and organisational engagement for the process to lead to

improved teamworking. Integration of TEAM into other

MDT assessment processes may enhance its potential to

facilitate improvements by providing leverage and incen-

tives for participation and improvement. In the UK it is

mandatory for all cancer teams to hold an annual meeting

to discuss their teamworking; TEAM could provide focus

and structure to these meetings, and could also be used as

supplementary evidence for the mandatory annual Peer

Review assessments, and also for team or individual job

appraisals. The NCAT, responsible for supporting imple-

mentation of cancer policy in England, funded the

development of TEAM through their MDT Development

Programme. Upon completion of the validation and testing

of TEAM its widespread use will be promoted by NCAT.

This is likely to include dissemination through the National

Development Programme (a biannual meeting attended by

senior cancer clinicians and managers across England)

and also through other streams of national work aimed

at improving patient care such as the National Patient

Experience programme and CONNECTED (national

communication skills training programme).

Following other health care specialties, like surgery and

anaesthesia, cancer care is increasingly focussed on

ensuring optimal teamwork and decision-making.7,17 To

ensure validity in such team assessment exercises, the tools

used to assess how a cancer team functions and how it can

improve ought to be scientifically robust and the process

clinically meaningful. Our preliminary work suggests that

TEAM meets these criteria and can make an important

contribution to improving teamwork in multidisciplinary

cancer teams, and by doing so will ultimately improve

patient care.

ACKNOWLEDGMENT We thank the team members who par-

ticipated in the development and testing of TEAM (as part of testing

MDT-FIT) and the Trust personnel who facilitated their involvement;

other affiliate members of Green Cross Medical Ltd who have sup-

ported this work; and the NCAT MDT Development steering group

and subcommittee members for their input and comments. Supported

in part by the NCAT. Sevdalis, Brown, and Lamb are also affiliated

with the Imperial Centre for Patient Safety and Service Quality,

which is funded by the National Institute for Health Research (NIHR).

REFERENCES

1. Saini KS, Taylor C, Ramirez AJ, et al. Role of the multidisci-

plinary team in breast cancer management: results from a large

international survey involving 39 countries. Ann Oncol. 2011;

23:853–9.

2. Kane B, Luz S, O’Briain DS, McDermott R. Multidisciplinary

team meetings and their impact on workflow in radiology and

pathology departments. BMC Med. 2007;5:15.

4026 C. Taylor et al.

3. Taylor C, Munro AJ, Glynne-Jones R, et al. 2010 Multidisciplinary

team working in cancer: where are we now? BMJ. 2007;340 c951.

4. Department of Health. Manual for cancer services. London:

Department of Health; 2004.

5. National Cancer Action Team. National Cancer Peer Review

Programme. Report, 2009/2010: an overview of the findings from

the 2009/2010 National Cancer Peer Review of cancer services in

England. London: National Cancer Action Team; 2010.

6. Lamb B, Brown K, Nagpal K, Vincent C, Green JSA, Sevdalis N.

Quality of care management decisions by multidisciplinary cancer

teams: a systematic review Ann Surg Oncol. 2011;18:2116–25.

7. Lamb BW, Green JSA, Sevdalis N. Decision-making in surgical

oncology. Surg Oncol. 2011;20:163–8.

8. Davies AR, Deans DA, Penman I, et al. The multidisciplinary

team meeting improves staging accuracy and treatment selection

for gastro-esophageal cancer. Dis Esophagus. 2006;19:496–503.

9. Coory M, Gkolia P, Yang I, Bowman R, Fong K. Systematic

review of multidisciplinary teams in the management of lung

cancer. Lung Cancer. 2008;60:14–21.

10. Burton S, Brown G, Daniels IR, Norman AR, Mason B, Cunn-

ingham D. MRI directed multidisciplinary team preoperative

treatment strategy: the way to eliminate positive circumferential

margins? Br J Cancer. 2006;94:351–7.

11. Haward R, Amir Z, Borrill C, et al. Breast cancer teams: the

impact of constitution, new cancer workload, and methods of

operation on their effectiveness. Br J Cancer. 2003;89:15–22.

12. Lanceley AS, Savage J, Menon U, Jacobs I. Influences on mul-

tidisciplinary team decision-making. Int J Gynecol Cancer. 2008;

18:215–22.

13. Kidger J, Murdoch J, Donovan JL, Blazeby JM. Clinical deci-

sion-making in a multidisciplinary gynaecological cancer team: a

qualitative study. BJOG. 2009;116:511–7.

14. Leo F, Venissac N, Poudenx M, Otto J, Mourox J; Groupe

d’Oncologie Thoracique Azureen (GOThA). Multidisciplinary

management of lung cancer: how to test its efficacy? J ThoracOncol. 2007;2:69–72.

15. Stalfors J, Lundberg C, Westin T. Quality assessment of a mul-

tidisciplinary tumour meeting for patients with head and neck

cancer. Acta Otolaryngol (Stockh). 2007;127:82–7.

16. Blazeby JM, Wilson L, Metcalfe C, Nicklin J, English R,

Donovan JL. Analysis of clinical decision-making in multidisci-

plinary cancer teams. Ann Oncol. 2006;17:457–60.

17. Lamb BW, Sevdalis N, Arora S, Pinto A, Vincent C, Green JS.

Teamwork and team decision-making at multidisciplinary cancer

conferences: barriers, facilitators, and opportunities for improve-

ment. World J Surg. 2011;35:1970–6.

18. Vincent C, Moorthy K, Sarker SK, Chang A, Darzi AW. Systems

approaches to surgical quality and safety: from concept to mea-

surement. Ann Surg. 2004;239:475–82.

19. Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N.

The impact of nontechnical skills on technical performance in

surgery: a systematic review. J Am Coll Surg. 2012;214:214–30.

20. Hull L, Arora S, Kassab E, Kneebone RL, Sevdalis N. Obser-

vational Teamwork Assessment for Surgery (OTAS): content

validation and tool refinement. J Am Coll Surg. 2011;212:234–

43.

21. Undre S, Koutantji M, Sevdalis N, et al. Multi-disciplinary crisis

simulations: the way forward for training surgical teams. World JSurg. 2007;31:1843–53.

22. McCulloch P, Rathbone J, Catchpole K. Interventions to improve

teamwork and communications among healthcare staff. Br JSurg. 2011;98:469–79.

23. Lamb BW, Sevdalis N, Mostafid H, Vincent C, Green JSA.

Quality improvement in multidisciplinary cancer teams: an

investigation of teamwork and clinical decision-making and

cross-validation of assessments. Ann Surg Oncol. 2011;18:3535–

43.

24. Jamtvedt G, Young JM, Kristoffersen DT, Thomson O’Brien

MA, Oxman AD. Audit and feedback: effects on professional

practice and health care outcomes. Cochrane Database Syst Rev.2003;(3):CD000259.

25. Taylor C, Ramirez AJ. Multidisciplinary team members’ views

about MDT working: results from a survey commissioned by the

National Cancer Action Team. London: National Cancer Action

Team; 2009. http://www.ncin.org.uk/mdt.

26. National Cancer Action Team. The characteristics of an effective

multidisciplinary team. London: National Cancer Action Team;

2010. http://www.ncin.org.uk/mdt.

27. Fleissig A, Jenkins V, Catt S, Fallowfield L. Multidiscpilinary

teams in cancer care: are they effective in the UK? Lancet Oncol.2006;7:935–43.

28. Abell N, Springer DW, Kamata A. Developing and validating

rapid assessment instruments. Oxford: Oxford University Press;

2009.

29. Spector PE. Summated rating scale construction. London: Sage

Publications; 1992.

30. Cortina JM. What is coefficient alpha? An examination of theory

and applications. J Appl Psychol. 1993;78:98–104.

31. Taylor C, Brown KB, Sevdalis N, Green JSA. Developing and

testing a novel, evidence-based and user-tested toolkit for

assessing and improving teamworking in multidisciplinary cancer

teams (abstract). Paper presented at: European Cancer Congress

Stockholm, September 25, 2011. http://new.ecco-org.eu/ecco_

content/StockholmAbstractbook/index.html.

32. Yun GW, Trumbo CW. Comparative response to a survey exe-

cuted by post, e-mail, and Web form. J Comp Mediat Commun.2000;6(1).

TEAM to Self-assess MDTs 4027