variability of likert scales effects on data collection results

VARIABILIT

Y OF L

IKERT SCALES

EFFECTS ON DATA

COLLECTIO

N RESULTS

C L A S S 7 – 2 0 1 1 / 2 0 1 2

MI M

E D - F M

UP

• Questionnaires• Likert scale: what is it?• Different Likert scales• Study objective• Methods

- Study participants - Study design

• Results• Conclusions

Questionnaries

Introduction Objective Methods

Results Conclusion

Currently, people’s opinion about an issue or service is a valuable way for researchers to acquire information in diverse areas of knowledge.

• A questionnaire is an essential instrument that aims to collect this information based on the inquisition of the population in study;• It consists in a certain amount of questions covering a topic of interest to investigators;• Is useful in almost every kind of job.

Likert Scale and Its Variables - A Comparative Study


Having in mind the need to measure and classify this information, one reaches the conclusion of the pressing necessity for scales, tools that permits measuring people’s attitudes and opinions[1].

Likert scale

1. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500.


Results Conclusion

LIKERT SCALE

A scale that consists in asking people to respond to a series of

statements about a topic, in terms of the extent to which they

agree or disagree with them.

What is it?

Measure pain, quality of life

Applications


2. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500. 3.Huber JF, H.J., Zumstein MD, Ruflin G, Lüscher M., Visual circle scale (VCS)-a patient-friendly scale to measure pain compared to VAS and Likert scale. Z Orthop Unfall, 2007. 145(6): p. 795-7.

[2]

[3]


Results Conclusion

DIFFERENT LIKERT SCALES

Different scales produce

different resultsThen, different

Likert scales may produce different

results

Search: some articles were found confirming our

idea


4 Dawes, J., Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 2008. 50(1): p. 61-77.

[4]


Results Conclusion

INFLUENCE OF A CENTRAL ANSWER

Different Likert scales

produce different results?

Does a neutral answer influence

the results?

Does both "I don't want to answer"

and neutral options influence the

results?Apparently not

studied yet.

Apparently not studied yet.


5. Kulas JT. , S.A., Middle category endorsement in odd-numbered Likert response scales: Associated item characteristics, cognitive demands, and preferred meanings. Journal of Research in Personality, 2009. 43: p. 489–49

Already studied

[5]

Does a "I don't want to answer" option

influence the results?


Results Conclusion

OBJECTIVES

Discover if the addition of a neutral element, a “refusal to answer” option or both

combined has repercussions in the results obtained.

Evaluate the reproducibility of the scales used in the questionnaire.


6. Hansson LM, G.M., Bergström R, Factors affecting reproducibility of dietary reports using food frequency questionnaires. Eur J Clin Nutr, 2000. 54(8).

[6]


Results Conclusion


Study Design:

One questionnaire with two versions

Questionnaire 1 Questionnaire 2Number of questions in

each questionnaire:

12Same questions in both

questionnaires

Methods


Results Conclusion

Type of study

Our study is included in a Longitudinal Study type.BECAUSE :

-There is a follow-up period of the individuals in study, which can be longer or shorter;

- Data is gathered at least in two points in time which allows us to observe changes in the population.



Results Conclusion

Students from classes 6 and 7

Study Participants:

All students that belong to the first year of the Integrated Master’s Degree in

Medicine of Faculdade de Medicina da Universidade do Porto, distributed by

classes 1 to 22

Exce

pt


Methods


Results Conclusion


Monday Tuesday Wednesday

Thursday Friday

8:30-11 h 1 4 6 9

14-16:30h 2 x 7 1016:30-

19h 3 5 8These ten blocks where randomly divided in two

groups with SPSS.

Questionnaire 1Questionnaire 2

Study Participants:

Methods


Results Conclusion

About the questionnaires:

• Theme of the questionnaire: Evaluation systems and study methods

Theme with less opinion change

probability


Why?

Methods


Results Conclusion

• Half of the participants answer to version 1 and the other half to version 2

• One month later, the first half answer to version 2 and the other half to version 1

Why?


To avoid certain biases.


Methods


Results Conclusion


• The questionnaires ought to be confidential.

How?The participants wrote their

names on the questionnaires and we

ensured confidentiality .

Why?In order to receive from

the participants their truly opinion


Methods


Results Conclusion



Comparison scale:I totally

agree I agree I disagree I totally disagree

… plus a neutral answer:Neither agree nor disagree

… plus refusal to answer:I don’t want to answer

Methods


Results Conclusion



Three types of comparison

Comparison scaleVs

Comparison scale plus neutral answer

Comparison scale:Likert scale with “I totally

agree”, “I agree”, “I disagree” and “I totally disagree”

Comparison scaleVs

Comparison scale plus refusal to answer option

Comparison scaleVs

Comparison scale plus neutral and refusal to

answer option

Methods


Results Conclusion

About the questionaires:


To evaluate the reproducibility

Comparison scaleVs

Comparison scale

Comparison scale plus refusal to answer

optionVs

Comparison scale plus refusal to answer

option


answer optionVs


answer optionComparison scale:

Likert scale with “I totally agree”, “I agree”, “I disagree”

and “I totally disagree” Methods


Results Conclusion

Questionnaire 1I1. Comparison Scale2. Comparison Scale3. Comparison Scale + neutral answer4. Comparison Scale5. Comparison Scale + refusal to answer6. Comparison Scale + neutral answer + refusal to answer7. Comparison Scale + refusal answer

II1. Comparison Scale + refusal to answer2. Comparison Scale + neutral answer + refusal to answer3. Comparison Scale4. Comparison Scale + neutral answer + refusal to answer5. Comparison Scale

Questionnaire 2I1. Comparison Scale + neutral answer2. Comparison Scale3. Comparison Scale 4. Comparison Scale + neutral answer5. Comparison Scale 6. Comparison Scale + neutral answer + refusal to answer7. Comparison Scale

II1. Comparison Scale + refusal to answer2. Comparison Scale

3. Comparison Scale + refusal to answer4. Comparison Scale 5. Comparison Scale + neutral answer + refusal to answer

Results

Introduction

Objective

Methods

Results

Conclusion

Study Participants:


250

234

158

REPRODUCIBILITY

Introduction

Objective

Methods

Results

Conclusion

Valid

(n)

Question

Scale

Proportion of

Agreement

[Confidence interval]

Kappa


Weighted Kappa


157 I.2O sistema de avaliação por

exames é justo.

Standard 0,76[0,68;0,84]

0,54[0,42;0,67]

0,60[0,49;0,71]

156 II.1O meu estudo

rende mais quando trabalho

em grupo.

Standard+NR 0,64[0,56;0,72]

0,47[0,36;0,58]

N.A.

157 I.6O número de

alunos por turma no 1º ano da

FMUP é excessivo.

Standard+NR+Neutral

0,47[0,39;0,55]

0,30[0,20;0,41]

N.A.

STANDARD SCALE VS. NEUTRAL OPTION SCALE

Introduction

Objective

Methods

Results

Conclusion

Valid n Question Answers’

Optionsn(%)

Standart Scale

n(%)Standard Scale

+ Neutral Option

Proportion of Agreement[Confidence

Interval]

155

I.4. É justo ser necessário ter frequência a uma disciplina (frequentar pelo menos 2/3) das aulas para ser admitido a exame teórico.

Discordo Totalmente 15 (9,7) 12 (7,7) 0.29 [0,12;0,52]Discordo 37 (23,9) 33 (21,3) 0.37 [0,24;0,52]

Não Concordo nem Discordo ------- 33 (21,3) -------

Concordo 79 (51,0) 57 (36,8) 0,37 [0,28;0,48]Concordo

Totalmente 24 (15,5) 20 (12,9) 0,26 [0,13;0,44]

156

I.1O sistema de avaliação contínua é vantajoso para os alunos.

Discordo Totalmente 3 (1,9) 2 (1,3) 0 [0;0,54]Discordo 25 (16,0) 30 (19,2) 0,41 [0,26; 0,58]

Não Concordo nem Discordo ------- 31 (19,9) -------


Totalmente 24 (15,4) 20 (12,8) 0,38 [0,22;0,56]

155I.3 Penso que o grau de exigência da FMUP é superior ao de outras.

Discordo Totalmente 0 (0) 1 (0,6) 0 [0;0,94]Discordo 7 (4,5) 3 (1,9) 0,43 [0,12;0,80]

Não Concordo nem Discordo -------- 12 (7,7) -------


Totalmente 102 (65,8) 82 (52,9) 0,64 [0,55;0,73]

STANDARD SCALE VS. REFUSAL TO ANSWER OPTION SCALE

Introduction

Objective

Methods

Results

Conclusion

Validn Question Answers’

Options

n(%)Standard

Scale

n(%)Standard Scale

+ Refusal to Answer Option

Proportion of Agreement [Confidence

Interval]

158

I.5 Os professores do 1º ano da FMUP estão sempre disponíveis para tirar dúvidas

Discordo Totalmente 1 (0,6) 0 (0,0) 0 [0;0]Discordo 28 (18,1) 30 (19,4) 0,35 [0,21;0,51]Concordo 113 (72,9) 87 (56,1) 0,54 [0,45;0,63]

Concordo Totalmente 13 (8,4) 29 (18,7) 0,14 [0,05;0,30]

Não respondo ------- 9 (5,8) -------

158

I.7 Quando há avaliação contínua, acho antipedagógico os alunos não irem tendo ao longo do semestre feedback do seu desempenho

Discordo Totalmente 0 (0,0) 3 (1,9) 0 [0;0]Discordo 12 (7,6) 11 (7,0) 0,15 [0,04;0,39]Concordo 90 (57,3) 70 (44,6) 0,47 [0,37;0,57]Concordo

Totalmente 55 (35,0) 64 (40,8) 0,49 [0,38;0,60]

Não respondo ------- 9 (5,7) -------

158

II.3 Tenho necessidade de recorrer a bebidas energéticas e/ou vitaminas para acompanhar o ritmo de estudo exigido pelo curso.

Discordo Totalmente 70 (44,3) 71 (44,9) 0,52 [0,41;0,62]Discordo 57 (37,1) 50 (31,6) 0,35 [0,25;0,47]Concordo 25 (15,8) 22 (13,9) 0,31 [0,17;0,48]

Concordo Totalmente 6 (3,8) 11 (7,0) 0,31 [0,10;0,61]

Não respondo ------- 4 (2,5) -------

STANDARD SCALE VS. BOTH OPTIONS COMBINED SCALE

Introduction

Objective

Methods

Results

Conclusion

Validn Question Answers’

OptionsN(%)

Standard Scale

N(%)Standard Scale + Both Combined Options Scale

Proportion of Agreement [Confidence

Interval]

158

II.2 O estudo é mais proveitoso quando trabalho num local

silencioso.

Discordo Totalmente

0 (0,0) 0 (0,0) 0,00 [0,00;0,00]

Discordo 8 (5,1) 4 (2,5) 0,09 [0,01;0,43]Não Concordo nem Discordo ------- 11 (7,0) -------


Totalmente 89 (56,3) 98 (62,0) 0,67 [0,57;0,75]Não Respondo ------- 0 (0,0) -------

158

II.4 Frequentemente acho mais rentável

estudar pelos apontamentos de

colegas mais velhos do que pela bibliografia

recomendada.

Discordo Totalmente

0 (0,0) 0 (0,0) 0,00 [0,00;0,00]

Discordo 13 (8,2) 8 (5,1) 0,31[0,12;0,59]Não Concordo nem Discordo ------- 11 (7,0) -------

Concordo 81 (51,3) 56 (65,4) 0.44 [0,34;0,55]Concordo


158

II.5 É difícil conciliar as actividades extra-

curriculares (por exemplo praticar um desporto, aprender

um instrumento musical, etc…) com o trabalho exigido na

FMUP

Discordo Totalmente

8 (5,1) 3 (1,9) 0,10 [0,01;0,46]

Discordo 29 (18,4) 26 (16,5) 0,25 [0,14;0,41]Não Concordo nem Discordo ------- 22 (13,9) -------



Conclusion

Introduction

Objective

Methods

Results

Conclusion

Freedom of choice

It depends…

VS Reproducibility


F A C U L D A D E D E M E D I C I N A D A U N I V E R S I D A D E D O P O R T O 2 º S E M E S T R E M A R Ç O D E 2 0 1 2

Abel Trzan

Ana Fonseca

Ana Filipa Ferreira

Ana Isabel Maduro

Catarina Tinoco

Guilherme Amorim

Joana Antunes

João Dias

José Carneiro

Maria João Bastos

Maria João Costa

Pedro Barros

Pedro Santos


variability of likert scales effects on data collection results

Documents

comparative study likert

idealikert scale

patientfriendly scale

number of scale points

visual circle scale

different resultsthen

different resultssearch

comparative studyhaving