misresponse to reversed and negated items in surveys: a review

Misresponse to Reversed and Negated Items in Surveys: A Review

Bert WeijtersHans Baumgartner

Misresponse to Reversed and Negated Items

Motivation Should reversed items be included in multi-item

summative scales?□ The use of reversed items in surveys often leads to

problems (low reliabilities, poor fit, distorted factor structure);□ Reversed items control for acquiescence, serve as cognitive

“speed bumps” and may encourage more complete coverage of the construct’s domain of content;

If reversed items are to be used, does it matter whether the reversal is achieved through negation or through other means?

What’s the link between reversal and negation, what types of MR result, what psychological mechanisms are involved, and how can MR be avoided?


Item reversal vs. item negation Authors often fail to draw a clear distinction between

reversals and negations and use ambiguous terms such as ‘negatively worded items’, which makes it unclear whether they refer to reversed or negated items, or both;

Examples from the Material Values scale (Richins and Dawson 1992):

□ It sometimes bothers me quite a bit that I can't afford to buy all the things I’d like.

□ I have all the things I really need to enjoy life.□ I wouldn't be any happier if I owned nicer things.


Empirical data we analyzed items from volumes 1 through 36 of

JCR (1974 till the end of 2009) and volumes 1 through 46 of JMR (1964 to 2009);

we included all Likert-type scales for which the items making up the scale were reproduced in the article and factor loadings or item-total correlations were reported;

total of 66 articles in which information about 1330 items measuring 314 factors was provided;

of the 1330 distinct items in the data set, 608 came from JCR and 722 from JMR;


Item negation Items can be stated either as an assertion (affirmation) or

as a denial (disaffirmation) of something (Horn 1989); Negation is a grammatical issue; Classification of negations in terms of two dimensions:

□ what part of speech is negated (how a word is used in a sentence: as a verb, noun/pronoun, adjective, adverb or preposition/conjunction);

□ how the negation is achieved (by means of particle negation, the addition of no, the use of negative affixes, negative adjectives and adverbs, negative pronouns, or negative prepositions);

Misresponse to Reversed and Negated ItemsNegated by means of

Part of speech

Not, n’t No negative affixes negative adjectives

and adverbs

negative pronouns

negative prepositions

Total

Verb [This salesperson does not make false claims.]

n.a. dislike, disagree, etc.

[I dislike food shopping very

much.]

reluctant, hesitant, never, rarely, seldom, hardly (ever), less, little, etc.

[I seldom daydream.]

n.a. without, instead of, rather than,

etc.[This supplier

sometimes promises to do things without actually doing

them later.]

135 (44.7%) 5 (1.7%) 26 (8.6%) 5 (1.7%) 171 (56.6%)

Noun/pronoun

not everyone, not (only), etc.

[I and my family will consume only certain brands, not

others.]

no object, no reason, no

purpose, etc.[Clipping,

organizing, and using coupons is

no fun.]

discomfort, disagreement,

etc.[There is

considerable disagreement as

to the future directions that this hospital should take.]

little, few, a lack of, none of the,

not much, neither of, etc.

[Many times I feel that I have little influence over things that happen to me.]

no-one, nobody, none, nothing,

etc.[Energy is really not my problem because there is simply nothing I can do about it.]

except for, without, with the

exception of, instead of, rather

than, etc.[American

people should always buy

American-made products instead

of imports.]

4 (1.3%) 17 (5.6%) 5 (1.7%) 10 (3.3%) 4 (1.3%) 14 (4.6%) 54 (17.9%)

Adjective n.a. n.a. uninterested, dishonest, etc.

[Most charitable organizations are

dishonest.]

rarely, less, etc.[I would be less loyal to this rep

firm, if my salesperson

moved to a new firm.]

n.a. n.a.

55 (18.2%) 4 (1.3%) 59 (19.5%)

Misresponse to Reversed and Negated ItemsNegated by means of

Part of speech

Not, n’t No negative affixes negative adjectives

and adverbs

negative pronouns

negative prepositions

Total

Adverb not much, etc.[After I meet

someone for the first time, I can

usually remember what they look like, but not much about them.

no longer, etc.[Hard work is no longer essential

for the well-being of society.]

[I often dress unconventionally even when it's likely to offend

others.]

rarely, less, etc.[I feel I have to do things hastily and maybe less

carefully in order to get everything

done.]

n.a. n.a.

2 (.7%) 1 (.3%) 3 (1.0%) 1 (.3%) 7 (2.3%)

Preposition/ Conjunction

not for, not (just) in, not (only) as, not until, not if (incl. unless),

not because, etc.[I enjoyed this

shopping trip for its own sake, not just for the items

I may have purchased.]

n.a. n.a. n.a. for nothing, etc.[I don't believe

in giving anything away for nothing.]

n.a.

10 (3.4%) 1 (.3%) 11 (3.7%)

Total 151 (50.0%) 18 (6.0%) 68 (22.5%) 41 (13.5%) 5 (1.7%) 19 (6.3%) 302


Item reversal an item is reversed if its meaning is opposite to a

relevant standard of comparison (semantic issue); three senses of reversal:

□ reversal relative to the polarity of the construct being measured;

□ reversed relative to other items measuring the same construct:

reversal relative to the first item reversal relative to the majority of the items

□ reversal relative to a respondent’s true position on the issue under consideration (Swain et al. 2008);


Item reversal (cont’d) in our data set of 1330 items, between 83 and 86

percent of items were nonreversed (depending on the definition of reversal);

the proportion of factors (or subfactors in the case of multi-factor constructs) that do not contain reversed items was 70 percent;

only 8 percent of factors (out of 314 factors) were composed of an equal number of reversed and nonreversed items (i.e., the scale was balanced);


Integrating item negation and item reversal

Item reversal

Non-reversed Reversed

Item negation

Non-negated

Regular (RG) items

Talkative, enjoying talking to people

Polar opposite (PO) items

Quiet, preferring to do things by oneself

Negated

Negated polar opposite (nPO) items

Not quiet, preferring not to be by oneself

Negated regular(nRG) items

Not talkative, not getting much pleasure chatting with people


Cross-classification of negation and reversal

Reversal relative to

Polarity of construct Polarity of first item Dominant keying direction Total

Non-reversed Reversed Non-

reversed Reversed Non-reversed Balanced Reversed

No negation(s) 71.1% 8.7% 71.0% 8.7% 70.3% 5.2% 4.2% 79.7%

Negation(s) 11.7% 8.7% 15.4% 4.9% 13.7% 2.9% 3.7% 20.3%

Total 82.8% 17.4% 86.4% 13.6% 84.0% 8.1% 7.9% 100.0%


Misresponse to negated and reversed items

Item Consistent responding

Misresponse to negated items

(NMR)

Misresponse to reversed items

(RMR)

Misresponse to polar opposites

(POMR)

Talkative (RG) A A A A

Not talkative (nRG) D A A D

Quiet (PO) D D A A

Not quiet (nPO) A D A D

MR → within-participant inconsistency in response to multiple items intended to measure the same construct;


Theoretical explanations of MR:Comprehension

Careless responding (Schmitt and Stults 1985):□ respondents fail to pay careful attention to individual

items and respond based on their overall position on an issue [RMR];

□ more likely when a reversed item is preceded by a block of nonreversed items;

□ Remedies: alert Rs to the presence of reversed items, avoid

tedious surveys, and be careful with involuntary Rs; increase Rs’ ability and avoid distractions; use balanced scales, alternate the keying direction, and

disperse the items;


Theoretical explanations of MR:Comprehension (cont’d)

Reversal ambiguity: Rs may not view antonyms as polar opposites [POMR]; contradictories vs. contraries:

Antonyms can be contradictories or contraries, depending on whether they are bounded or unbounded (Paradis and Willners 2006);

Negations of the same core concept are clearly contradictories, but things are more complicated when the core concept differs;

simultaneous disagreement is more likely when items are worded extremely (McPherson and Mohr 2005);

“Buddhism’s ontology and epistemology appear to make East Asians relatively comfortable with apparent contradictions” (Wong et al. 2003, p. 86) [RMR];


Theoretical explanations of MR:Comprehension (cont’d)

□ Remedies: use more sophisticated procedures to identify

appropriate antonyms (formulate linguistic contrasts in two stages; see Dickson and Albaum 1977);

may be particularly useful in cross-cultural research; bounded antonyms have to be pretested and

unbounded antonyms have to be used with care; extreme statements should be avoided;


Theoretical explanations of MR:Retrieval

Item-wording effects:□ Confirmation bias (Davies 2003; Kunda et al. 1993);□ Directly applicable to antonymic reversals;□ For negation reversals, confirmation bias can lead to

MR if a non-negated polar opposite schema is readily available (Mayo et al. 2004);

□ Remedies: Use polar opposite reversals to get richer belief samples,

even though they may increase apparent MR; Negation reversals have few retrieval benefits;


Theoretical explanations of MR:Retrieval

Positioning effects:□ Dispersed PO items reduce carryover effects and can

increase coverage, but the task is more taxing for Rs and internal consistency may suffer;

□ Item similarity may determine whether Rs engage in additional retrieval when items are grouped together;

□ Remedies: The use of dispersed antonyms should encourage the

generation of distinct belief samples; Avoid very similar (negated) statements when items are

grouped;


Theoretical explanations of MR:Judgment

Item verification difficulty (Carpenter and Just 1975; Swain et al. 2008):□ MR is a function of the complexity of verifying the

truth or falsity of an item relative to one’s true beliefs, which depends on whether the item is stated as an affirmation or negation [NMR];

□ Remedies: Negations are problematic because they increase the

likelihood of making mistakes (remember there are many types of negations);

Negated polar opposites are most error-prone; Mix of regular and PO reversals should be best;


Item verification difficulty

Truth valueTrue False

Negation

Affirmation

MR


Theoretical explanations of MR:Response

Acquiescence: Rs initially accept a statement and subsequently re-consider it based on extant evidence; the first stage is automatic, the second requires effort (Knowles and Condon 1999) [RMR];

Remedies: □ Although response styles are largely individual

difference variables, situational factors may be under the control of the researcher (e.g., reduce the cognitive load for Rs);

□ Problems with online surveys;


Theoretical explanations of MR:Response (cont’d)

Asymmetric scale interpretation: the midpoint of the rating scale may not be the boundary between agreement and disagreement for Rs (esp. if the response categories are not labeled; cf. Gannon and Ostrom 1996) [RMR];

Remedies:□ Use fully labeled 5- or 7-point response scales;


Consequences of MR Effect on scale reliability: [based on 4 studies]

RG . .68nRG .60PO .57nPO .49

Effect on correlations between items measuring the same construct:

□ when content saturation is high, social desirability is low, and endorsement frequencies are moderate, the correlation between the four types of items can be quite high, although the findings showed evidence of POMR (Jackson and Lay 1968);

□ Ahlawat (1985) showed that the correlations between nPO and other items were particularly low;


Consequences of MR (cont’d) Effect on factor structure: [based on three studies]

□ The factor structure can emerge clearly when the items are high in content saturation and free of social desirability, but RMR and POMR are apparent;

□ In Schriesheim and Eisenbach (1995), RG items performed best and nPO items did very poorly;

Effect on correlations with other scales:□ Holden et al. (9185) showed that item negation was

more detrimental to criterion-related validity than item reversal;


Example item with 4 negationsTop management in my company has let it be known in no uncertain terms that unethical behaviors will not be tolerated.

misresponse to reversed and negated items in surveys: a review

Documents

use of reversed items

distinct items

worded items

negated itemsnegated

negated items1motivationshould

negated itemsitem reversal

jmr misresponse

negated itemsempirical