mgto 324 recruitment and selections scale and test construction kin fai ellick wong ph.d. department...

MGTO 324 Recruitment and Selections

Scale and Test Construction

Kin Fai Ellick Wong Ph.D.Department of Management of Organizations

Hong Kong University of Science & Technology

Prologue

• In the last lesson, I have discussed the scientific elements in testing

• Today, we focus on how a test can be constructed

• In particular, you are expected to understand the following– The concepts of “item”, “scale”, and “test”– Different item formats– How to write a set of good items – Multiple-item scaling

Outline


Part I: Basic concepts

Part II: Item Formats

Part III: Writing good items

Part IV: Multiple-item scaling

Part I: Basic Concepts

• What is a (psychological) test?– Measurement device or technique

• To quantify behavior or aid in the understanding and prediction of behavior

– A set of items designed to measure characteristics of human beings that pertain to behavior


• What is an item?– A specific stimulus to which a person responds overtly

• What is the English word of “邂逅”• You can manage well in interpersonal relationship

– Overt behaviors (Scientific Standard)• Observable• Measurable• Can be replicated

1 2 3 4 5

Strongly disagree

Strongly agree


• What is a scale?– The quantified scores obtained from a test– The raw scores are related to some defined theoretical or

empirical distribution– The matching between the raw score and the theoretical

meaning of that score• E.g., 0oC = freezing point; 100oC = boiling point

– The same theoretical meaning could be represented by different scales

• Temperature: Degree Celsius vs. Degree Fahrenheit• Length: Meter vs. Feet; Kilometer vs. Mile• Weight: lb vs. kg• Wealth: HK$ vs. US$


• What is a scale?– Examples

• Thermometer– Raw scores = 2.4 cm;

– Degree Celsius = 100

– Theoretically (empirically) = boiling point

• HKCEE– Raw scores = 87

– Grade = A

– Theoretically (empirically) = Excellent students

• IQ test– Raw scores = 2400

– IQ = 130

– Theoretically (empirically) = Gifted individual


• Essential steps in scale and test construction– Have a clear definition of what (i.e., the psychological construct) the test is supposed

to measure• E.g., Locus of control; self-efficacy

– Generating a set of items that at least seems to capture the construct– Determining the scale format – Pilot tests to assess the face validity, reliability, construct validity, and criterion

validity– Deleting or revising less useful items– Assessing reliability and validity again– Revising again– Assessing again– Revising again– Assessing again– Revising again – Assessing again– Shorten the scale– Assessing again….

Outline







Item Formats

Dichotomous formatDichotomous formatE.g., True/FalseE.g., True/False

Polytomous formatPolytomous formatE.g., Multiple-choiceE.g., Multiple-choice

Likert-type formatLikert-type format

Cumulative (Guttman) formatsCumulative (Guttman) formats


• Dichotomous formats– Offers two alternatives for each item

• True/False; or Select a more appropriate statement– “You often spend more than three hours on typing every day”

» Agree vs. Disagree

– “Generally speaking, the salary for me accurately represents my contribution to the organization”

» Agree vs. Disagree

– “Which job, technical or administrative, do you prefer?”

» Technical vs. Administrative

– Scores • Simply count the number of items a person endorse

– Commonly used in both educational and personality tests


• Dichotomous formats– Some famous scales used dichotomous formats

• Locus of control (Rotter, 1966)– “the degree to which people believe they are masters of their own fat

es” from the OB textbook (Robin, 2003)– High in externality

» Less satisfied with jobs; Higher absenteeism rates; More alienated from the work; Less involved in their jobs

– Choose one» A. Many of the unhappy things in people’s lives are partly due to bad luck» B. People’s misfortunes result from the mistakes they make

– Choose one» Heredity plays the major role in determining one’s personality» It is one’s experiences in life which determine what one is like


• Dichotomous formats– Advantages

• Simple, easy to administer and score• Absolute judgment; people should declare one of the two

alternatives

– Disadvantages• Effect of memorizing materials• Requires numerous items to produce reliable results

(chance = 50%)


Item Formats






• Polytomous formats– Offers more than two alternatives (e.g., usually 4 to 5)

• One correct choice and other distractors• Multiple-choice

– Scores• Number of items correctly answered

– How many distractors?• More distractors may not be better than less distractors (Sidick, Barrett, &

Doverspike, 1994)

– Less influenced memory effects (relative to dichotomous format)– Mainly used in educational tests


Item Formats






• Likert-type formats– Usually, people are required to express their degree of dimensi

on on a statement• I can manage well in interpersonal relationship

– Monotone items• A higher score suggests higher agreement

1 2 3 4 5

Strongly disagree

disagree neutral agree Strongly agree


• Likert-type formats– Number of alternatives

• Odd (5, 7, or 9) • Even (6, or 10): avoid midpoint

– Popular in psychological tests• It can be subject to various psychometric analyses, such as factor analy

sis• For example, the General Self-efficacy Scale• Self-efficacy

– “The individual’s belief that he or she is capable of performing a task” from OB textbook (Robin, 2003)

– Higher self-efficacy» Less likely to give up under difficult situations» Usually perform better than

http://userpage.fu-berlin.de/~health/engscal.htm


Item Formats






Cumulative (Guttman) formatsCumulative (Guttman) formats– Items on the same dimension are set up in ascending order– Subject with a particular attitude will agree with all items on one

side of that position and disagree with other items– Example Addition, long-division, and calculus– Monotone items


Cumulative (Guttman) formatsCumulative (Guttman) formats– Advantages

• A single score carries complete information about the response patterns– Calculus OK = division OK = addition OK– Marriage OK = being neighbor OK– When there is no random error

• Provides a test of the unidimensionality of what are to be tested– Cumulative response pattern will not be obtained when the items do not me

asure only one dimension

– Disadvantages• Problems resulted from random error• Difficult to find domains that are unidimensional

– Less popular than Likert-type format

Outline







Define clearly– Clearly define what you want to measure– Check the face and content validity

• Is the items out of syllabus• How many psychological factor I want to measure? List them all

I want to develop a test that helps me hire employees who have strong self-learning tendency

What do you want to measure from the test? What is “self-learning tendency”? Give a clear definition before moving to the next step


Clearly think about the item formats– Think carefully what type of tests as well as what statistical anal

yses you want to use– Some statistical analyses may not be appropriate for certain ite

m formats• Rank order is not appropriate for t test• Polytomous responses may not be able to be analyzed by Factor Analys

is

– Likert-type or dichotomous format seems to be a good choice as the default


Sources of items– Discourse and text

• From brainstorming or informal conversation• Asking others the meaning of “self-learning”

– Use qualitative method is a more systematic method• Open-ended interview or question• Focus group• Content analysis

– Newspaper– Classic literature

– The goal is to generate a pool of items that seems to measure what we want to measure

• E.g., “When I have problems, I’ll seek help from books prior to people.”


Nature of items– The meanings of items should be clear, straightforward, and can be easily

understood• Avoid exceptionally long items• Avoid reading difficulty

– Don’t use jargon….

• Social desirability– “I like self-learning”

• Offensiveness– Pay attention to the problems of sexism and racism

• Avoid double-barreled items– “seeking help from people is a better method than seeking help from books because

people are more accessible.

• Reverse items– “Most knowledge could not be learned without the help from others”– “I can learn almost all knowledge by myself”


Generate an item pool– You first need to generate a set of item pool– The numbers of items in this pool are usually much more than

the final version of the scale– Do a preliminary test

• Select useful items• Discard or revise other items (you will learn these skills later in this

course)

– You may need several rounds of revision before the scale becomes reliable and valid


What should be included in a test apart from the basic items?– Clear instruction for subjects

• Give examples whenever possible

– Questions for demographic information• Age, gender, education level, etc.

– Declare that how the collected information will be used• Is it confidential? The purpose of collecting the data; who will assess the

data, etc.

Outline







True score theory– Due to the matter of precision, measures may vary from time to

time– The true scores can hardly be obtained by only one measure– Sometimes it may overestimate or underestimate the true

score, the errors are assumed to be randomly distributed– The true scores can be obtained by averaging multiple

responses


The concepts of multiple-item scaling– I want to measure one’s attitude toward whether secondary

students can have romantic relationships– Single item

• I encourage secondary students to develop romantic relationship

– Two items• I encourage secondary students to develop romantic relationships• We should respect the secondary students’ freedom of engaging in

romantic relationships

– The issues of reliability and validity


Advantages multiple-item scaling– It allows testing the nature of a construct

• A set of items may capture more than one dimension– Intelligence:

» Memory span, verbal ability, and visual-spatial ability

• Items measuring same dimension or same construct should be highly inter-correlated

• Items measuring different dimensions are supposed to have relatively low inter-item correlation

• It allows us to check scale validity through Factor analysis– Discuss again later

– Increase reliability and validity• Remember the concept of true score theory• Random errors may exert impacts on a particular single response• Effects of random errors can be “neutralized” by measuring multiple responses

mgto 324 recruitment and selections scale and test construction kin fai ellick wong ph.d. department...

Documents

concepts of item

different item formats

behavior slide

outline scale

psychological test

selections scale

set of items

basic concepts essential