Measurement, Reliability and Validity

Fundamentals of Measurement

The relationship between Reliability and Selection

The relationship between Validity and Selection

How Validity is applied in utilizing assessments

7-3

Key Concepts

• Measurement

• The process of assigning numbers to objects to

represent quantities of an attribute of the objects

• Scores

• The amount of the attribute being assessed

• Correlation between scores

• A statistical measure of the relation between the two

sets of scores

The Role of Measurement

Assists the selection

manager with understanding

an applicants attributes

It involves numbers playing

an important role

It is essential to the

implementation and

administration of selection

programs

7-5

Importance and Use of Measures• Summary of measurement process

• (a) Choose an attribute of interest

• (b) Develop operational definition of attribute

• (c) Construct a measure of attribute as operationally defined

• (d) Use measure to actually gauge attribute

• Results of measurement process

• Scores become indicators of attribute

• Initial attribute and its operational definition are transformed into a numerical expression of attribute

The Role of Measurement

Criteria

Measurement of an

employees success on

the job

Dollar of sales or dollars

saved in context of money

or the targets achieved

Predictors

Forms of assessment

used to measure criteria

Interviews , Performance

Tests, Paper pencil Tests,

or Computer Tests.

The Role of Measurement

Predictors:

Background

information

Resumes

Applications

Interviews

Tests:

Aptitude or Ability

Achievement

Personality

Use of Measures in Staffing

7-8

7-9

Measurement: Standardization

• Involves

• Controlling influence of extraneous factors

on scores generated by a measure and

• Ensuring scores obtained reflect the attribute

measured

• Properties of a standardized measure

• Content is identical for all candidates measured

• Administration of measure is identical for all

candidates

• Rules for assigning numbers are clearly specified

and agreed on in advance

7-10

Measurement: Differences in

Objective and Subjective Measures

• Objective measures

• Rules used to assign numbers to attribute

are predetermined, communicated, and

applied through a system

• Subjective measures

• Scoring system is more elusive, often

involving a rater who assigns the numbers

The Role of Measurement

Evaluating Criteria:

Will situation change

alter the criteria

Is it realistic and

represent the job

Does management

accept the criteria

The Role of Measurement

Evaluating Predictors:

Does the predictor

appear appropriate

Is the cost less than an

inaccurate decision

Has it been

standardized

Can it be administrated

to a group v/s an

individual

7-13

Quality of Measures

• Reliability of measures

• Validity of measures

• Validity generalization

Concept of Reliability

Reliability Defined

The degree of dependability,

consistency, or stability of

scores on measures used in

selection

Concept of Reliability

Factors Influencing The

Reliability of a Assessment

Method of Estimating Reliability

Individual Differences Among Respondents

Length of a Measure

• Test Question Difficulty

• Administration of a measure

7-16

Quality of Measures: Reliability

• Definition: Consistency of

measurement of an attribute

• A measure is reliable to the extent it

provides a consistent set of scores to

represent an attribute

• Reliability of measurement is of

concern

• Both within a single time period and

between time periods

• For both objective and subjective measures

7-17

Ex. 7.6: Summary of Types of

Reliability

7-18

Quality of Measures: Reliability

• Measurement error

• Actual score = true score + error

• Deficiency error: Occurs when there is failureto measure some aspect of attribute assessed

• Contamination error: Represents occurrence of unwanted or undesirable influence on the measure and on individuals being measured

7-19

Ex. 7.7 - Sources of Contamination Error

and Suggestions for Control

7-20

Quality of Measures: Reliability

• Procedures to calculate reliability estimates

• Coefficient alpha

• Should be least .80 for a measure to have an acceptable degree of reliability

• Inter-rater agreement

• Minimum level of inter-rater agreement - 75% or higher

• Test-Retest reliability

• Concerned with stability of measurement

• Level of r should range between r = .50 to r = .90

• Intra-rater agreement

• For short time intervals between measures, a fairly high relationship is expected - r = .80 or 90%

7-21

Quality of Measures: Reliability

• Implications of reliability

• Standard error of measurement

• Since only one score is obtained from an

applicant, the critical issue is how accurate the

score is as an indicator of an applicant’s true

level of knowledge

• Relationship to validity

• Reliability of a measure places an upper limit on

the possible validity of a measure

• A highly reliable measure is not necessarily valid

• Reliability does not guarantee validity - it only

makes it possible

7-22

Quality of Measures: Validity

• Definition: Degree to which a measure truly

measures the attribute it is intended to

measure

• How well an assessments predicts employee

performance.

• Accuracy of measurement

• Exh. 7.9: Accuracy of Measurement

• Accuracy of prediction

• Exh. 7.10: Accuracy of Prediction

Concept of Validity

Types of Validity Strategies

Criterion-related Validity

testing the relationship between the predictor and criterion in relation to job performance

Content Validity

Validates those job behaviors that are necessary for successful job performance

Construct Validity

Test validation method that determines whether a test measures certain traits or qualities that are important in performing the job

7-24

Ex. 7.9: Accuracy of

Measurement

7-25

Exh. 7.10: Accuracy of

Prediction

7-26

Exh. 7.10: Accuracy of

Prediction

7-27

Validity of Measures in Staffing

• Importance of validity to staffing process

• Predictors must be accurate representations of characteristics to be measured

• Predictors must be accurate in predicting job success

• Validity of predictors explored through validation studies

• Two types of validation studies

• Criterion-related validation

• Content validation

7-28

Ex. 7.12: Concurrent and Predictive

Validation Designs

7-29

Ex. 7.12: Concurrent and Predictive

Validation Designs

7-30

Content Validation

• Content validation involves

• Demonstrating the questions/problems (predictor scores) are a representative sample of the kinds of situations occurring on the job

• Criterion measures are not used

• A judgment is made about the probable correlation between predictors and criterion measures

• Used in two situations

• When there are too few people to form a sample for criterion-related validation

• When criterion measures are not available

• Exh. 7.14: Content Validation

7-31

Validity Generalization

• Degree to which validity can be extended to other contexts

• Contexts include different situations, samples of people and time periods

• Situation-specific validity vs. validity generalization

• Exh. 7.16: The Logic of Validity Generalization

• Distinction is important because

• Validity generalization allows greater latitude than situation specificity

• More convenient and less costly not to have to conduct a separate validation study for every situation