monitoring the quality of an italian public psychiatric service: a...

Eur. J. Psychiat. Vol. 19, N.° 1, (5-18)2005

Key words: Quality assurance, Psychiatric servi-ces, Satisfaction, Functioning, Quality of life, Clini-cal status.

Monitoring the quality of an Italian publicpsychiatric service: A four dimensions study of the outcome

Antonio Amatulli*Paolo Azzone*,**Debra Srebnik***

* Ospedale G. Salvini, GarbagnateMilanese - Milan, Italy

** Scuola di Specializzazione in Psichiatria,Università degli Studi di Milano - Milan, Italy.

*** Department of Psychiatry andBehavioral Sciences - University ofWashington, Washington, D.C., USA

ABSTRACT – Objective: Within the concept of the outcome of hea1th services, the user’ssatisfaction has to be integrated by other more objective measures of health and quality oflife. Debra Srebnik and coworkers have proposed a Survey for Monitoring the Quality ofPublic Mental Hea1th Services (SMQPMHS), which covers the following dimensions:Satisfaction, Functioning, Quality of Life and Clinica1 Status. The research main goalswere as follows: a) to study the psychometric properties of the Ita1ian version of the Sur-vey, and b) to study possible differences between the outcome of psychiatric patients appl-ying to our outpatient facilities vs. the American sample.

Methods: The Italian version of SMQPMHS was proposed to all patients receiving careat 2 Italian outpatient psychiatric facilities over a period of 2 months. 291 subjects accept-ed to participate in the study and fi1led in adequately the questionnaire.

Results: Mean scores of the 13 variables of the Survey were very similar to thoseobserved in the Arnerican sample. A matrix of correlations between each variable andeach of the others indicated adequate internal consistency. A principal component analysissupported the four-dimensions model of the Survey. Patients recently hospitalized showeda poorer Clinical Status and a lower Satisfaction. Patients unemployed at the time of thesurvey showed a poorer Clinical Status.

Conclusions: The Italian version of SMQPMHS showed good psychometric proper-ties, even though concurrent validity needs further study. No differences emerged betweenthe present sample and the American sample.

Introduction

Measurement and evaluation of whatmental health services do for people is moreand more important at all levels of clinicalpractice (Goldman et al. 1990). Administra-tors demand performance reports for theirsystem of budget management; serviceproviders can no longer consider them-selves being (and being worth) only whatthey do; consumers are growing more andmore aware of having the right to rate thecare they receive on its merits; and themembers of the families of the patients actas custodians or guarantors of all theseaspects. Too often, health care professionalshave failed to seek the participation of theclients in choices that are inherent to theircare, and this often means that they madedecisions “in the patient’s exclusive inter-est” without really knowing what that inter-est is! (Wallace et al. 1999).

From the point of view of the users ofthe mental health service, obtaining theirperspective on care quality contributes totheir sense of freedom of choice, autonomyand empowerment. Therefore growingemphasis is being placed in mental healthcare, on the use of the perceptions anddegree of satisfaction of the users in theevaluation of the quality of the care provid-ed (Holcomb et al. 1998) (Davis & Fong1996, Ruggeri et al. 2000, Parasuraman etal. 1988, Rosenthal & Shannon 1997). Fol-lowing these precepts, many instrumentshave been developed and used, both forpurposes of research and for the purpose ofreorganizing service processes.

It is a well-known experience that ques-tionnaires specifically designed for the ratingof satisfaction tend to receive a generallypositive range of responses, thus there is stillmuch work to be done, at the level of devel-

oping methods and procedures for patients torate quality of the care and assistance provid-ed (Cochrane 1972, Zastovny et al. 1995).

One of the dimensions most commonlytaken into account in evaluation of the out-come and quality of treatments is the degreeof consumer satisfaction (Khayat & Salter1994, Aharony & Strasser 1993, Ross et al.1995, Sitzia & Wood 1997). In Italy eventhe national and regional health plansemphasize the consumer’s satisfaction as akey principle orienting service organizationand the selection of treatments whichshould be made available to the patients.

While a user-oriented service does needmanagement, the assessment of the percep-tion and the degree of the patient’s satisfac-tion also involves challenging clinical andmethodological issues (Cleary et al. 1991).For instance, Risser has identified certaindimensional aspects of user satisfaction, suchas costs, the personal qualities of the treatingphysician and the nature of the interpersonalrelationship, the professional skill of thephysician and the perceived quality of thetreatment received (Risser 1975). Ware et al.(1983) pointed out other related factors [e.g.,interpersonal relations (respect, courtesy,sympathy), technical aspects of the treatment(its appropriateness and accuracy), accessi-bility of treatment (logistics, waiting peri-ods), financial aspects, effectiveness and out-come, continuity of treatment and others].

It is therefore quite clear that whatevermeans are used to investigate the perceptionand degree of satisfaction of the user, it isessential to assess all the areas involved.

Finally, it is important to determine howthe services can make use of the patients’perceptions to improve the effectiveness oftreatment and the quality of the careprocesses. Patient perspectives can be usedin reviewing or verifying and reengineering

6 ANTONIO AMATULLI ET AL.

both organization and management (Cleary& Edgman-Levitan 1997, Barnes et al.1999). Thus it is to be hoped that health carewill proceed to operate with a view to con-tinuous improvement of the quality of ser-vice, based on periodic multidimensionalassessments.

Assessments of the perceptions and/ordegree of satisfaction of the patients (forexample in a mental health center) shouldinclude an expanded view of the functions ofthe person and the sphere in which he lives.For example, the conceptual model relativeto the outcomes of psychiatric patients pro-posed by Rosemblatt and Attkisson may beuseful in overcoming a one-dimensionalreading of the performance of the user.These authors have proposed four areas ofevaluation of the outcome: psychopatholog-ical and symptomatological, personal andsocial functioning, the degree of satisfactionand the quality of life (Rosemblatt &Attkisson 1993). This means that appropri-ate predictors have to intercept and correlatewith these areas. This model was used bySrebnik and her colleagues to develop out-come indicators that would “draw” in a suf-ficiently complete manner the dimensionsof quality relative to a mental health service(Srebnik et al. 1997).

We decided to repeat the study in Italy.We first translated the instrument to Italian,then back-translated it to English, and theauthors reviewed it for accuracy.

The main goals of the present study canbe summarized as follows:

1. To assess whether the psychometricproperties of the English version, and par-ticularly the internal consistency and theunderlying factor structure had been pre-served in the Italian one.

2. To compare satisfaction data from thestudy of Srebnik et al. of users of MentalHealth Centers (MHCs) in the United Stateswith data from users of Italian MHCs. Infact, both settings studied offered mentalhealth service on an outpatient basis, but anumber of peculiarities of the Italian psy-chiatric system might have been the basisfor a greater or lower satisfaction: a) a widerrole of psychiatrists in the Italian MHC,including the functions covered by non clin-ical case managers in the US psychiatricfacilities, b) an extremely rigid job marketc) a very limited availability of shelteredhouses, and d) the specific cultural frame-work the patients were living in, possiblydetermining different social tolerancetowards the expression of hostility or insat-isfaction towards health institutions.

Methods

Setting

The study was carried out in two outpa-tient psychiatric structures in the PsychiatricUnit of Garbagnate Milanese, classified oneas a MHC and the other as a PsychiatricOutpatient facility. Both structures offered,at the time of collection of the data, healthand psycho-social services to patientsaffected by mental illness, resident in sevenmunicipalities in the north of the provinceof Milan. The area of our Health Care Unitincluded both suburban and rural zones.

Together, the two centers served 1836people. The services provided were as fol-lows: psychiatric visits, psychological inter-views, psychotherapy, psychological tests,house visits, nursing activities in the centersor in the patients’ homes, psycho-socialactivities for the purpose of helping the

MONITORING THE QUALITY OF AN ITALIAN PUBLIC PSYCHIATRIC SERVICE... 7

patients to re-socialize or work, and consul-tation by social workers.

Measurements

The Italian translation (made by theauthors) of the Survey for Monitoring theQuality of Public Mental Health Services(SMQPMH) consists of a self-administeredquestionnaire that includes a total of 45items, from which 12 measurements aredrawn, organized into 3 domains or areas,supplemented by a case manager rated Clin-ical Status scale. The following 4 domainsare therefore covered by the client self-report and clinician-rated instruments: Sat-isfaction, Functioning, Quality of Life andClinical Status.

The domain of Satisfaction measures thedegree of satisfaction of the users with theirinvolvement in the therapeutic programme,with the appropriateness of the therapy andsecurity relative to their visits to the mentalhealth center. The Functioning area mea-sures the physical, mental and social func-tioning of the patients and their ability tomanage their lives and their symptoms. Thedomain of the Quality of Life measuressafety in the home, achievement of goalsand experiences of violence. Finally, Clini-cal Status, rated by clinicians, is measuredby symptoms, functioning, substance abusecondition and compliance with treatment.

The survey was developed to measure theoutcome of psychiatric treatment. It wasdrawn up in such a way as to include theaspects of outcome that are most importantfor the different stakeholders of psychiatricservices, such as users, members of theirfamilies, professionals and administrators.The questionnaire was drawn up makingample use of existing instruments designedto measure separate dimensions of the out-

come of psychiatric treatments. Specifical-ly, the survey included the eight-item ClientSatisfaction Questionnaire (CSQ, Nguyenet al. 1983), the SF-12 (Bogaert-Martínez etal. 1996, McHorney et al. 1993), sevenitems from the Lehman Quality of LifeInterview (Lehman 1991), and four itemsfrom a California Public Mental Health Sur-vey (Veit 1995). The other items weredeveloped by Srebnik and coworkers. Thecase manager survey consisted of the Four-Dimensional Classification Scale (Comtoiset al. 1994), which includes four single-item7-point scales that assess symptoms, func-tioning, substance abuse and treatment com-pliance. The original study, carried out inthe United States on 236 users of outpatientmental health services, showed that theSMQPMH scales representing each dimen-sion showed adequate internal consistencyand evidence of concurrent validity. Fur-thermore, the analysis of the main compo-nents confirmed the 4 dimensional modelthat underlies the instrument.

Subjects

The questionnaire was submitted to all theusers who had consecutively received healthcare or psychosocial services from the twosites of the study during a period of twomonths (June-July 2000), provided that wasnot their first contact with the centers. Out of317 patients receiving the questionnaire, 309agreed to fill it out and return it. Filling it outtook between 10 and 25 minutes; the person-nel at the Center were available to provide alittle help when necessary. At the same time,the treating physician filled out the ClinicalStatus component. After collecting the ques-tionnaires, we added the personal data andthe number of specialized contacts thepatients had received in the six previousmonths.


Since some of the questionnaires wereonly partly completed, we included in thestudy only those questionnaires that con-tained enough information to establish atleast 8 measurements out of 13.

Of the 291 users who satisfied this condi-tion about half were women and about half(46%) were married. The average age was43.35 years (SD = 15.1). Fewer than 20% ofthe patients had finished high school andfewer than 2% had graduated from universi-ty. The rather low social and cultural level ofthe sampling was confirmed by the verylimited number of professionals and self-employed. Among other things, more than athird of the sample did not perform any pro-ductive activity.

The diagnoses were mainly Affective Psy-chosis (31.7%) and Schizophrenic Disorders(30.6%), but the sample also includedpatients with Neuroses (17.0%) or Personali-ty Disorders (11.4%). Compared to theprevalence of the various diagnoses in thegeneral population of patients of our Service,significant differences were observed onlyfor Schizophrenic Disorders and OrganicMental Disorders. Specifically, our sampleshowed a higher prevalence of SchizophrenicDisorders (30.63% vs. 21.02%, X2 = 12.59, p= 0.0004) and a lower prevalence of OrganicDisorders (3.32% vs. 8.99%, X2 = 10.01, p= 0.0016).

Data analysis

The internal consistency of the Italianversion of the questionnaire was assessedthrough a matrix of correlations betweeneach of the measurements of outcome andeach of the others, both those belonging tothe same domain and those belonging to dif-ferent domains. The assignment of severalmeasurements to the same domain in the

survey implies the assumption that thesemeasurements represent underlying aspectsthat are analogous and interdependent. Weexpected, therefore, that the correlationsamong measurements belonging to the samedomain would be stronger than thoseobserved among different domains.

In order to determine whether the underly-ing factorial structure of the survey remainedstable in the Italian version as well, we alsoperformed a principal components analysis.The reliability of each measurement of thescale was examined using Cronbach’s alphacoefficient. Lastly, the concurrent validity ofthe values of the 4 areas of the survey wasverified by analysis of variance, using as fac-tors of classification the state of employmentand the existence of episodes of hospitaliza-tion in specialized environments.

Results

Average values of theQuestionnaire in the populationinvestigated

Table I presents the average values of eachmeasurement of the sampling. A comparisonwith the values obtained by Srebnik and hercollaborators applying the SMQPMH to theusers of a number of territorial psychiatricservices in the U.S. shows considerable con-vergence between the two populations.However, the values relative to physical andmental functioning cannot be directly com-pared, as the American authors report valuesobtained after standardization of the data.

Internal consistency of the scale

Table II shows the correlations betweeneach of the measurements of outcome and


each of the remaining measurements belong-ing both to the same domain and to differentdomains. All the reciprocal correlationsbetween measurements of the domain ofSatisfaction and measurements of thedomain of Functioning are significant with amedian of correlation coefficients of 0.49(range 0.34 - 0.71 vs. range 0.24 - 0.58 in theAmerican sample) for the domain of Satis-faction and 0.55 (range 0.34 - 0.71 vs. range0.17 - 0.60 in the American sample) for thatof Functioning. Among measurementsbelonging to the area of the Quality of Life,the correlations observed were not as strong,with only three statistically significant val-ues out of six, with a median of correlationcoefficient of 0.14 (range -0.01 - 0.34 vs.range -0.08 - 0.27 in the American sample).That might be due to a “floor” effect, ascriminal offences are relatively rare and con-

sequently most patients had no victimizationor safety issue.

On the whole, the correlation among mea-surements belonging to different domainsappeared less strong than among measure-ments belonging to the same domain. Themeasurements of Functioning and Satisfac-tion showed a number of significant correla-tions, but the association appeared lessstrong than that which was observed withineach of the two domains, with a median cor-relation coefficient of 0.23 (range 0.09 - 0.37vs. range: -0.05 - 0.21 in the American sam-ple). Half of the correlations between mea-surements of the domain of the Quality ofLife and of Functioning were significant(range -0.11 - 0.54 vs. -0.11 - 0.30 in theAmerican sample), while the relationshipwith the domain of Satisfaction appearedslight (range 0.02 - 0.31 vs. -0.11 - 0.30 inthe American sample).


Table IMean scores of the survey on Satisfaction, Functioning, Quality of Life and Clinical Status of 273 users ofGarbagnate CMHC vs. the survey of 236 American users reported by Srebnik et al. (1997)

Italian sample American SampleDomain and Number Mean SD Mean SD Scoremeasurement of item range

SatisfactionClient Satisfaction Questionnaire 8 3.21 .55 3.29 .61 1-4Involvement in treatment 2 4.22 .85 3.98 1.03 1-5Treatment appropriateness 2 4.13 .96 3.98 1.07 1-5Safety at the mental health center 1 .89 .31 .89 .32 0-1

FunctioningPhysical 6 61.33 25.67 10.04 .74 0-100Mental 6 47.09 25.82 10.03 .66 0-100Social and leisure 3 4.53 1.35 4.74 1.28 1-7Skills for handling stress and symptoms 4 3.28 .85 3.54 .87 1-5

Quality of lifeSafety 4 4.75 1.13 4.89 1.30 1-7Concerns about living condition 2 1.68 .37 .60 .38 1-2Goal attainment 4 .75 .26 .73 .18 0-1Victimization 1 1.95 .17 1.82 .31 1-2

Clinical statusFour-Dimensional Classification Scale 4 4.63 1.18 4.14 1.05 0-6

MONITORING THE QUALITY OF AN ITALIAN PUBLIC PSYCHIATRIC SERVICE... 11Ta

ble

IIC

orre

latio

n m

atri

x of

dat

a fo

r 13

out

com

e m

easu

res

in 4

dom

ains

obt

aine

d fr

om a

sur

vey

of 2

73 u

sers

of

the

CM

HC

in G

arba

gnat

e

Satis

fact

ion

Func

tioni

ngQ

ualit

y of

life

Dom

ain

and

mea

sure

men

t11

23

45

67

89

1011

12

Satis

fact

ion

1. C

lient

Sat

isfa

ctio

n Q

uest

ionn

aire

(.76

)2.

lnvo

lvem

ent i

n tr

eatm

ent

.60‡

3. T

reat

men

t app

ropr

iate

ness

.71‡

.57‡

4. S

afet

y at

the

men

tal h

ealth

cen

ter

.40‡

.34‡

.37‡

Func

tioni

ng5.

Phy

sica

l (.7

9).1

2.0

9.1

8*.1

7*6.

Men

tal (

.82)

.30‡

.20†

.24‡

.21†

.62‡

7. S

ocia

l and

leis

ure

(.80

).3

4‡.1

5*.2

8‡.2

6‡.3

4‡.5

4‡8.

Ski

lls f

or h

andl

ing

stre

ss a

nd

sym

ptom

s (.

80)

.37‡

.16*

.30‡

.27†

.53‡

.71‡

.5

5‡Q

ualit

y ol

life

9. S

afet

y (.

81)

.31‡

.29‡

.30‡

.31‡

.40‡

.44‡

.5

4‡.3

5‡10

. Con

cern

s ab

out l

ivin

g co

nditi

on.1

5*.0

8.1

5*.1

1.1

2.2

5‡.3

3‡.2

0†.3

4‡11

. Goa

l atta

inm

ent

.03

.04

.03

.02

-.11

.04

.07

-.09

-.01

.12

12. V

ictim

izat

ion

.03

.02

.06

.12

-.00

.06

.11

.05

.17*

.15*

.05

Clin

ical

sta

tus

13. F

our-

Dim

ensi

onal

C

lass

ific

atio

n Sc

ale

(.74

).2

5‡.1

8§.2

4‡.1

2*.2

5‡.2

9‡.2

3‡.3

5‡.1

9†.1

7§.1

6*.1

2*

(1 ) f

or m

esur

es c

onta

inin

g m

ore

than

2 it

ems

Cro

nbac

h’s

αis

giv

en in

par

enth

eses

.(*

) p

< 0

.05

(§)

p <

0.0

01(†

) p

< 0

.005

(‡)

p <

0.0

01

On the whole, the results of the matrix ofcorrelations seem to confirm the structureof the questionnaire and are in line with thedata reported by the American study of val-idation of the instrument. With respect tothat study, it is possible to observe correla-tions tending on the average to be strongerbetween the area of Clinical Status and theother areas (ranges: 0.12 - 0.25, 0.23 - 0.35,and 0.12 - 0.19 for Satisfaction, Function-ing and Quality of Life respectively; vs.0.13 - 0.30, -0.05 - 0.06 and 0.10 - 0.14 inthe American sample). On the other hand,the values of r observed testify to the exis-tence of a significant correlation betweenClinical Status and the other components ofoutcome, but indicate also that Clinical Sta-tus explains only a limited part of the vari-ability observed.

It seems to us that these values might beexplained in part by the higher level of com-petence of the clinicians in our survey, all ofthem being physicians specialized in Psy-chiatry, as compared with the context of theAmerican survey, in which bachelor levelcase-managers most often completed theClinical Status Component.

Factor analysis

The results of the principal componentsanalysis are shown in Table III. In order tocheck the structure of the instrument withinthe sample used for the study, we selected asolution with 4 factors. Each factor showedeigenvalues higher than 1. Assigning eachmeasurement to the factor for which themeasurement presented the highest factorloading value, the 4 measurements of thedomain of Functioning could be assigned tofactor 1, the measurements of Satisfactionto factor 2 and the Clinical Status to factor4. The measurements of the area of the

Quality of Life exhibited a more complexbehavior. The measurements of concernsabout Living conditions and Victimizationpresented higher factor loadings on factor 3,while Safety and Goals Attainment present-ed higher factor loadings on factors 1 and 4respectively. Similar problems were alsofound for the factor relative to the Quality ofLife in the study by Srebnik and collabora-tors. We agree with the American authorsthat reasons of clarity of interpretation sug-gest the inclusion of these last two measure-ments in the Quality of Life domain.

Global outcome indicators

The Clinical Status component yielded asingle global score representing the wholedomain of mental illness. Global indicatorsof Satisfaction, Quality of Life and Function-ing were obtained by taking the mean of thedifferent measurements of each domain, afterthe separate measures have been standard-ized around a mean of 10 and a standarddeviation of 1. For the domain of Satisfactionthe mean thus obtained was 10.03 ± 0.77.The internal reliability, measured usingCronbach’s alpha coefficient, was 0.76. Forthe domain of Functioning, the global out-come indicator was found to be 10.01 ± 0.83with a Cronbach’s alpha coefficient of 0.56.On the average, the patients included in thesampling presented a global indicator of theQuality of Life of 10.05 ± 0.57 with a Cron-bach’s alpha coefficient of 0.27.

Measurements of concurrentvalidity

As was done in the study by Srebnik andcollaborators, the validity of the SMQPMHwas tested in terms of concurrent validity.Thus, we studied the relationship among the


MONITORING THE QUALITY OF AN ITALIAN PUBLIC PSYCHIATRIC SERVICE... 13Ta

ble

III

Prin

cipa

l Com

pone

nt A

naly

sis

(Nor

mal

ized

Var

imax

Rot

atio

n) o

f 13

mea

sure

men

t of

outc

ome

with

com

pone

nt lo

adin

gs o

n fo

ur o

utco

me1

Dom

ain

and

mea

sure

men

t Fu

nctio

ning

Satis

fact

ion

Qua

lity

of li

feC

linic

al S

tatu

s

Satis

fact

ion

Clie

nt S

atis

fact

ion

Que

stio

nnai

re.1

9(-

.30)

.87

(.40

).0

3(-

.16)

-.02

(.06

)In

volv

emen

t in

trea

tmen

t.0

1(-

.23)

.82

(.35

)-.

05(-

.23)

.11

(-.0

3)T

reat

men

t app

ropr

iate

ness

.16

(-.0

9).8

2(.

33)

.07

(-.0

9)-.

03(.

16)

Safe

ty a

t the

men

tal h

ealth

cen

ter

.21

(.01

).6

3(.

33)

.10

(.01

)-.

11(-

.13)

Func

tioni

ngPh

ysic

al.7

8(.

35)

.01

(.15

)-.

17(-

.35)

.25

(-.0

9)M

enta

l.8

7(.

42)

.13

(.34

).0

1(-

.10)

.06

(.21

)So

cial

and

leis

ure

.71

(.30

).2

0(.

33)

.20

(.05

)-.

21(.

01)

Skill

s fo

r ha

ndlin

g st

ress

and

sym

ptom

s.8

1(.

40)

.20

(.28

).0

9(-

.22)

.17

(.12

)Q

ualit

y of

life

Safe

ty.6

1(.

03)

.18

(.31

).1

8(.

14)

-.14

(-.6

2)C

once

rns

abou

t liv

ing

cond

ition

.36

(.12

).0

5(.

11)

.60

(.50

)-.

32(-

.38)

Goa

l atta

inm

ent

.00

(.11

).1

0(.

08)

.11

(.53

)-.

74(.

54)

Vic

timiz

atio

n-.

06(-

.03)

.01

(.27

).8

3(.

37)

.09

(.01

)C

linic

al s

tatu

sFo

ur-D

imen

sion

al C

lass

ific

atio

n Sc

ale

.25

(-.1

8).2

1(.

19)

.42

(.20

).5

8(.

31)

Eig

enva

lues

4.15

(2.0

2)1.

88(3

.01)

1.29

(1.4

7)1.

09(.

94)

% V

aria

nce

acco

unte

d fo

r.2

5(.

16)

.21

(.23

).1

0(.

11)

.09

(.07

)

1D

ata

from

the

orig

inal

stu

dy b

y Sr

ebni

ket

al.

(199

7) a

re g

iven

in p

aren

thes

es.

values of the global outcome indicatorsemerging from the survey and two measuresrepresentative of dimensions commonlybelieved to be associated with a good out-come of mental health treatment. Thedimensions considered were the absence ofhospitalization for psychiatric problemsduring the study period and occupation in aproductive job.

As regards hospitalization for psychiatricproblems, we compared the averages of theglobal indicators of outcome of patients hos-pitalized during the study period with theaverages of patients who had not been hospi-talized during the same period. We expectedthat patients who had been hospitalizedwould present a generally poorer outcomewith respect to at least a few indicators. Theanalysis of variance (Table IV) showed a sig-nificant difference for two dimensions: Sat-isfaction and Clinical Status. In both cases,the patients who had been hospitalized dur-ing the study period presented outcomes that

were worse than those of the patients whowere not hospitalized.

With respect to the condition of employ-ment, we compared the outcome indicatorsof unemployed patients or invalids with thoseof employed or retired patients. We expectedthe outcome indicators for patients who hadperformed productive work to be better thanthose of patients who had not been able towork during the study period. This differencewas effectively observed for only one of theoutcome indicators, Clinical Status. As wasforeseeable, employed patients had a betterClinical Status than unemployed ones, andthe difference was statistically highly signifi-cant (Table V). On the whole, these resultsconfirm the concurrent validity of the Clini-cal Status component and in part of the Satis-faction global outcome indicator. We thinkthese data indicate that a more comprehen-sive concurrent measurement of outcome isnecessary for a better assessment of thevalidity of the survey as a whole in our worksetting.


Table IVImpact of hospitalization on global outcome indicators (ANOVA)

Non Hospitalized HospitalizedIndicator n. Mean SD n. Mean SD F (d.f.) p

Satisfaction 200 10.13 .66 33 9.40 1.01 28.74 (1,231) .000000Functioning 167 10.02 .79 24 9.91 1.10 .36 (1,189) .55Quality of life 180 10.06 .55 27 9.93 .68 1.23 (1,205) .27Clinical status 228 4.76 1.09 43 3.96 1.35 17.92 (1,269) .00003

Table VImpact of employment condition on 4 global outcome indicators (ANOVA)

Employed UnemployedIndicator n. Mean SD n. Mean SD F (d.f.) p

Satisfaction 148 10.09 .78 85 9.93 .74 2.33 (1,231) .13Functioning 123 10.01 .80 68 10.00 .90 .01 (1,189) .93Quality of life 134 10.09 .56 73 9.97 .58 2.29 (1,205) .13Clinical status 170 4.86 1.03 101 4.25 1.30 18.60 (1,269) .00002

Outcome of patients in relationto mental health services andthe services they provide

In order to study the effect of the servicesprovided by our mental health structures onthe outcome of the patients of those struc-tures, we examined the relationships betweenthe outcome indicators provided by theSMQPMH and the interventions effectivelyreceived by the users. We took into consider-ation 4 types of services: psychiatric visits,rehabilitation in day care facilities, nursingcare and the actions of social workers. Underthe heading of psychiatric visits we consid-ered the number of contacts received in thesix months prior to collection of the data. Onthe average, the patients who participated inthe study had been seen 4.09 ± 3.69 times bya specialist during the period reviewed. Asregards nursing care, rehabilitation in daycare facilities and actions by social workers,we divided the patients into two groups,depending on whether or not they hadreceived one or more of these services duringthe period of the study.

Since the attention and the activity ofpsychiatric services tend to concentrate onpatients with more severe symptoms inpsychopathological terms, we expectedthat the patients who had received moreservices would be characterized by a worseClinical Status. We expected, however, thatthe patients who had been seen by a spe-cialist more times or had received a broad-er range of services would present a betteroutcome, or at least a higher level of Satis-faction.

The correlations between the number ofpsychiatric visits received and each of theglobal outcome indicators did not give statis-tically significant results. The results of theanalysis of variances confirmed that thepatients who received a broader range of ser-

vices had a worse Clinical Status (F = 7.75,df = 1,269, p = 0.0058 for social worker’sinterventions; and F = 15.52, df = 1,269; p = 0.001 for nursing care), but did notshow any effect of these actions on theother outcome indicators, particularly onSatisfaction. Indeed, the patients whoreceived more attention from the socialworker or more day care rehabilitationshowed a lower level of Satisfaction thanthe other patients (F = 24.52, df = l,23l,p = 0.00000l for social worker’s interven-tions; and F = 6.39, df = l,23l, p = 0.012 forday care rehabilitation). This could per-haps be explained by the discrete level ofcorrelation observed between the measure-ments of the domain of Satisfaction andthose of the domain of Clinical Status.Since in our sample, as we have alreadypointed out, the more seriously ill patientstend to be less satisfied, it is likely that thesubgroup of patients who received socialand rehabilitation services, being charac-terized by a more serious Clinical Status,would tend to perceive the cares receivedin a more negative manner.

Discussion

The Italian version of the SMQPMH hasproved to be an easily administered, practi-cal instrument for studying differentaspects of the outcomes of patients treatedby our outpatient services. Its psychometricproperties have been found to be globallyconserved in spite of its ease of use andsimplicity. Furthermore, administering it inour service gave results that are for themost part comparable with those of theoriginal American study, thus confirmingits reliability.


To the basic format, we then added indi-cators of utilization of the service in order tofurther investigate the effect of the psychi-atric help performed by our mental healthstructures on the outcome of the patients.For this purpose, we took into consideration4 types of interventions: psychiatric andpsychological visits, rehabilitation in daycare facilities, nursing care and the actionsof a social worker. The results of this furtherinvestigation showed contrasting results thatwill have to be confirmed by a future repeti-tion of the study, which we have planned.

It is important, however, to report that,although it appeared plausible to us toexpect a higher level of Satisfaction on thepart of patients who had received more care,we did not find this to be the case. In actualfact, these first results would appear to indi-cate that a poorer Clinical Status tends tocharacterize users who are subjected to awider range of care, and for this reason theirlevel of Satisfaction is lower. This seems tocreate an intrinsic limit to the studies onuser satisfaction: either they will have to begeared to a less severely ill segment of thepopulation (which is in any case numerical-ly predominant) that uses the services, orthey are likely to give very misleadingresults, exactly as we have just said. In addi-tion, scores that are in any case “flattened”upward could lead to the dangerous andgroundless inference that the cares providedare all of good or high quality.

As evaluations of this kind are rathercommon, it was necessary to decide how todeal with this type of difficulty of assess-ment. For example, the authors of theSMQPMH themselves recently describedprocedures of “risk adjustment”, designedto “weigh” the scores achieved against theseverity of the symptoms shown (Hendryxet al. 1999). Similarly, one could also sim-ply group scores by clinical severity level.

This is an avenue of study and research inthe field of the evaluation of perceptionsand user satisfaction that we consider ofgreat importance in terms of method.

In relation also to these problems, theSMQPMH certainly had advantages: theyinclude the multidimensional aspect of eval-uation, in which the values of satisfactionare expressed with the other aspects of theoutcome. The further quantification that weadded regarding the extent of utilization ofthe service resources by each patient pro-vides additional help in this direction. Someauthors, however, point to other limits thatmust be understood and possibly dealt withwhen studies on perceptions and outcomesof treatments are used in the evaluation ofthe processes or quality of the cares provid-ed. In fact, these studies fail to intercept animportant (and often severely ill) group ofpatients: persons who suffer from psychi-atric problems but do not have access to theservice. Or all those individuals who leavethe out or inpatient treatment circuit prema-turely (Stockdill 1992).

In the last analysis, we can point to anumber of factors that make the SMQPMHand its Italian translation an extremely use-ful instrument in the assessment of thepatients’ perceptions and of additional out-come indicators.

First, there is its extremely manageableaspect (the small number of items –only 45in all for the four parameters– that differen-tiate it from most of the existing question-naires) and its ease-to-use.

Second, there is the possibility that it canbe used for more general evaluation of dif-ferent aspects of outcome, as well as foridentifying “critical areas” of the processesof care and treatment, which can then bemodified.


Finally, comparisons can be made betweendata drawn in different services (with the pre-caution of the variable “weight” of the preva-lence of different classes of disorder). TheSMQPMH, which in any case has shown, inthe Italian translation, results that are in mostpart comparable with those of the originalAmerican version, can prove to be anextremely useful instrument with a view tocontinuous quality improvement.

Its use is simple and inexpensive, repeti-tion of data collection is therefore not toocomplex and even just by comparing theaverage values of the scales some types ofevaluation (changes, critical aspects) appearextremely simple. Repetition of data collec-tion in time will make it possible to monitornumerous outcome indicators (but also indi-cators of processes and structures) that lie atthe base of a system for monitoring thequality of a community based psychiatricservice.

Finally, our service has just undertaken asecond data collection programme using thesurvey again. There are also plans for fur-ther studies of outcome validation in otherparts of Italy.

References

Aharony L, Strasser S. Patient satisfaction: What weknow about and what we still need to explore. Med CareRev 1993; 50(1): 49-79.

Barnes J, Stein A, Rosemberg W. Evidence-based medi-cine and evaluation of mental health services: Methodolog-ical issues and future directions. Arch Dis Child 1999;80(3): 280-285.

Bogaert-Martinez E, Caen E, Wilson W, et al. The SF-36 as a measure of functioning and health-related qualityof life in individuals with severe and persistent mental ill-ness: psychometric properties and normative data. Paperpresented at the Annual Conference on State Mental

Health Agency Services Research and Program Evalua-tion, Alexandria, VA, 1996.

Cleary PD, Levitan SE, Roberts M et al. Datawatch.Patients evaluate their hospital care: a national survey.Health Aff 1991; 10(4): 254-267.

Cleary PD, Edgman-Levitan S. Health care quality.Incorporating consumer perspectives. J Am Med Assoc1997; 278(19): 1608-1612.

Cochrane AL. Effectiveness and efficiency. London:Nuffield Provincial Hospitals Trust, 1972.

Comtois KD, Ries R, Armstrong HE. Case manager rat-ings of the clinical status of dually diagnosed outpatients.Hosp Community Psychiatry 1994; 45: 568-573.

Davis D, Fong M. Measuring outcomes in psychiatry:an inpatient model. Jt Comm J Qual Improv 1996; 22(2):125-133.

Goldman L, Francis Cook E, Oran J et al. Researchtraining in clinical effectiveness: Replacing “in my experi-ence...” with rigorous clinical investigation. Clin Res 1990;38(4): 686-693.

Hendryx MS, Dyck DG, Srebnik D. Risk-adjusted out-come models for public mental health outpatient programs.Health Serv Res 1999; 34(1Pt 1): 171-195.

Holcomb WR, Bertman BD, Hernme CA et al. Use of anew outcome scale to determine best practices. PsychiatrServ 1998; 49(5): 583-585, 595.

Khayat K, Salter B. Patient satisfaction surveys as amarket research tool for general practices. Br J Gen Pract1994; 44(382): 215-219.

Lehrnan AF. Quality of Life Interview Core Version.Baltimore: University of Mary land, Center for MentalHealth Services Research, 1991.

McHorney C, Ware J, Raczek A. The MOS 36-item SF-36: II. Psychometric and clinical tests of validity and mea-suring physical and mental health constructs. Med Care1993; 31: 247-263.

Nguyen T, Attkisson C, Stegner B. Assessment ofpatient satisfaction: development and refinement of a ser-vice evaluation questionnaire. Eval Program Plann 1983;6: 299-314.

Parasuraman A, Zethrnal VA, Barry LC. SERVQUAL:A multiple-item scale for measuring consumer perceptionsof service quality. J Retailing 1988; 64: 12-40.

Risser N. Development of an instrument to measurepatient satisfaction with nurses and nursing care in primarycare settings. Nursing Research 1975; 24(1): 45-52.


Rosemblatt A, Attkisson CC. Assessing outcomes forsufferers of severe mental disorder: A conceptual frame-work and review. Eval Program Plann 1993; 16: 347.

Rosenthal GE, Shannon SE. The use of patient percep-tions in the evaluation of health care delivery systems. MedCare 1997; 35(11 Suppl.): NS58-68.

Ross CK, Steward CA, Sinacore JM. A comparativestudy of seven measures of patient satisfaction. Med Care1995; 33(4): 392-406.

Ruggeri M, Lasalvia A, Dell’ Agnola R et al. Develop-ment, internal consistency and reliability of the VeronaService Satisfaction Scale-European Version. Br J Psychi-atry 2000; 177(suppl. 39): 41-48.

Sitzia J, Wood N. Patient satisfaction: a review of issuesand concepts. Soc Sci Med 1997; 45(12): 1829-1843.

Slade M, Thornicroft J, Glover G. The feasibility of rou-tine outcome measures in mental health. Soc PsychiatryPsychiatr Epidemiol 1999; 34(5): 243-249.

Srebnik D, Hendryx M et al. Development of outcomeindicators for monitoring the quality of public mentalhealth care. Psychiatr Serv 1997; 48(7): 903-909.

Stockdill J. A government manager’s view of mentalhealth advocacy groups. Adm Policy Ment Health 1992;20: 45-55.

Thornicroft G, Tansella M. Translating ethical princi-ples into outcome measures for mental health servicesresearch. Psychol Med 1999; 29: 761-767.

Veit S. California Mental Health Performance OutcomeProject. Report for the California State Department ofMental Health. Sacramento, CA: Department of MentalHealth, 1995.

Wallace T, Robertson E, Millar et al. Perceptions of careand services by the clients and families: a personal experi-ence. J Adv Nurs 1999; 29(5):1144-1153.

Ware JE, Snyder MK, Wright R et al. Defining and mea-suring patient satisfaction with medical care. Eval Pro-gram Plann 1983; 6(3-4): 247-263.

Williams B. Patient satisfaction: A valid concept? SocSci Med 1994; 38(4): 509-516.

Zastovny TR, Stratmann WC, Adams EH et al. Patientsatisfaction and experience with health services and quali-ty of care. Qual Manag Health Care 1995; 3(3): 50-61.

Address of correspondence:Dott. Paolo AzzoneVia San carlo 131/b20017- Rho (MI) Fax. +39029309903ITALYe-mail: [email protected]


monitoring the quality of an italian public psychiatric service: a...

Documents