03/17/1503/17/15

RESEARCH IN RESEARCH IN INFORMATION SYSTEMS INFORMATION SYSTEMS

MANAGEMENTMANAGEMENT(IMS 603)(IMS 603)

Topic 7:Topic 7:Measurement in ResearchMeasurement in Research

IntroductionIntroductionMeasurement in research consists of assigning numbers to empirical events in compliance with a set of rules.

1)Selecting observable empirical events

2)Using numbers or symbols to represent aspects of the events

3)Applying a mapping rule to connect the observation to the symbol

03/17/1503/17/15

Introduction (cont.)Introduction (cont.)Example 1:

To study people whom attend a computer exhibition at PWTC where all of the computer’s new models are on display. You are interested in learning the male-to-female ratio among visitors of the exhibition. You observe those enter the exhibition area.

• Record male as ‘m’ and female as ‘f’ or

• Record male as ‘1’ and female as ‘2’.

03/17/1503/17/15

Introduction (cont.)Introduction (cont.)Example 2:

To measure the opinion of people on several new computer models. This can be achieved by interviewing a sample of visitors and assign their opinions to scales ranging from Strongly Agree (1) … Neutral (3) … to Strongly Disagree (5).

03/17/1503/17/15

What is measured?What is measured?Concepts used in research may be classified as:

Objects •Include the things of ordinary experience such as people, automobiles, food etc.

Phenomena•Things that are not concrete such as attitudes, perception, opinion, satisfaction etc.

Properties•Characteristics of the objects

What is measured? (cont.)What is measured? (cont.)• A person’s physical properties may be stated

in terms of weight, height, posture.• Psychological properties include attitudes

and intelligence.• Social properties include leadership, ability,

class affiliation or status.

Rules of MeasurementRules of MeasurementA rule is a guide that instructs us on what to do. An example of a rule of measurement might be:•Assign the numerals 1 through 7 to individuals according to how productive they are. If the individual is an unproductive worker with little output, assign the numeral 1.•If a study on office computer systems is not concerned with a person’s depth of experience but defines people as users or nonusers, a ‘1’ for experience with the system and a ‘0’ for non experience with the system can be used.

Levels of MeasurementLevels of MeasurementVariables can be further differentiated in terms of the ‘level’ or nature of measurement that are ‘continuous’ or ‘discrete’ in their form.

Continuous variables•Have an infinite number of values that flow along a continuum.•On a continuum, values can be divided and sub-divided indefinitely in mathematical theory.•Even a five-point scale could be divided into a larger number of smaller units by sub-dividing between each pair of points on the scale.

Levels of MeasurementLevels of MeasurementDiscrete variables• Have relatively fixed set of separate values or

variable attributes.• Instead of a smooth continuum of values,

discrete variables contain distinct categories (eg. Gender: Male and Female)

Measurement LevelsMeasurement LevelsContinuous and discrete variables yield four levels of measurement (degree of precision of measurement).

The four levels of measurement are:

1.Nominal

2.Ordinal

3.Interval

4.Ratio

Measurement LevelsMeasurement Levels

Discrete / Categorical(Frequency)

Continuum / Continuous/

Scale(Score)

Nominal

Ordinal

Interval

Ratio

Categories with no order.

Categories with some order.

Arranges objects according to their magnitudes in units of equal interval.

Arranges objects according to their magnitudes in units of equal interval & has a true zero point.

Nominal ScaleNominal Scale• The simplest type of scale.• A scale in which the numbers of letters assigned

to objects serve as labels for identification or classification.

GENDER Males : 1

Females : 2

RACE Malays : 1

Chinese : 2

Indian : 3

Ordinal ScaleOrdinal Scale• A scale that arranges objects or alternatives

according to their magnitudes.• A typical ordinal scale, example to rate services,

brands, and so on as ‘excellent’, ‘good’, ‘fair’, or ‘poor’.

• We know ‘excellent’ is higher than ‘good’ but we do not know by how much nor would we know whether the gaps between ranks are the same or different.

Interval ScaleInterval Scale• A scale that not only arranges objects according

to their magnitudes, but also distinguished this ordered in units of equal interval.

• Example 1: Ratings of radio programs would involve program evaluations using a five- or seven-point scale.

• Hence, it would be possible not only to determine which program was best liked, second best liked, third best liked, etc. but also the amount by which one program was more liked than another.

Interval Scale (cont.)Interval Scale (cont.)• Example 2: If a temperature is 90 degree

Celsius, it cannot be said that it is twice as hot as 45 degree Celsius.

• The reason for this is that 0 degree Celsius does not represent the lack of temperature but a relative point on the Celsius scale.

• Due to the lack of an absolute zero point, the interval scale does not allow the conclusion that 90 is twice as great as the number 45, only that the interval distance is two times greater.

Ratio ScaleRatio Scale• At the ratio level, it is possible to measure the

extent to which one variable exceeds another on a particular dimension, and in addition, the scale of measurement has a true zero point.

• Example: when measuring distance in meters, zero means no distance at all. It is an absolute and non-arbitrary zero point.

• When measuring money in currency values, again zero means no money at all. The absolute zero point is an important factor because such scales also have exactly equal intervals between the separate points on the scale.

Criteria for good measurementCriteria for good measurement1. Reliability

The degree to which measures are free from error and therefore yield consistent results.

The reliability of a measure indicates the stability and consistency with which the instrument measures the concept.

Example: imperfections in the measuring process that affect the assignment of scores or numbers in different ways each time a measure is taken, such as a respondent who misunderstands a question are the cause of low reliability.

Criteria for good measurementCriteria for good measurement2. Validity

Is a test of how well an instrument that is concerned with whether we measure the right concept.

There are two type of validity: Internal and external validity.

Internal validity: concerned about issue of the authenticity of the cause-and-effect relationships

External validity: concerned about issue of the generalizability to the external environment.

03/17/1503/17/15

Goodness of Measures Goodness of Measures

1. Item Analysis

Test whether items in the instruments should belong there. Steps:

1. Calculate Total Score2. Divide respondents into high and

low score3. Compute t-test for each item 4. Use only items that are significant

2. Reliability Analysis

Is the measure without bias (error free) and therefore consistent across time and across items in the instrument? i.e. is it stable and consistent?

3. Validity Analysis

Is the instrument measuring the concept it sets out to measure and not something else?

03/17/1503/17/15

Goodness of MeasuresGoodness of Measures

GOODNESS OF DATA

Reliability (Accuracy)

Validity (Actuality)

Stability

Consistency

Test-retest

Parallel form

Interitem consistency

Split-halfLogical

(content)

Criterion related

Congruent (construct)

Face

Predictive

Concurrent

Convergent

Discriminant

03/17/1503/17/15

Reliability and ValidityReliability and Validity

Valid but UnreliableValid & Reliable Reliable but NOT

Valid

03/17/1503/17/15

ReliabilityReliabilityObserved scores may reflect true scores, Observed scores may reflect true scores, but it may reflect other factors as well:but it may reflect other factors as well:

stable characteristics: two people having the stable characteristics: two people having the same opinion may circle different responsessame opinion may circle different responsestransients personal factors such as moodtransients personal factors such as moodsituational factors, time pressure, time situational factors, time pressure, time variations in administration and mechanical variations in administration and mechanical factorsfactors

Reliability: Stability and consistencyReliability: Stability and consistency StabilityStability – over time, conditions, state of – over time, conditions, state of

respondentsrespondents ConsistencyConsistency – Homogeneity of times; items can – Homogeneity of times; items can

measure the construct independentlymeasure the construct independently

03/17/1503/17/15

Reliability of MeasuresReliability of MeasuresRELIABILITY

Stability Consistency

Test-retest Parallel form

Repeated measures on the same respondent; high correlation – high reliability

Two comparable sets of measures for the same construct; same items, same response format but different wording; Analysis - correlation

Interitem Split-half

Consistency of respondents’ answer to all the items; high correlation among responses to the items – Cronbach α

Correlation between two-halves of a measure; correlation between the two halves

03/17/1503/17/15

ValidityValidityMultiple indicators: - often used to capture a Multiple indicators: - often used to capture a given construct e.g. attitude; to given construct e.g. attitude; to cover the domain of the constructcover the domain of the construct robust - reduce random errorrobust - reduce random error Cronbach alpha - measures intercorrelation Cronbach alpha - measures intercorrelation

between indicators - they should be positively between indicators - they should be positively correlated but not perfectly correlatedcorrelated but not perfectly correlated

Construct ValidityConstruct Validity Face validityFace validity Convergent validity (Correlation to assess it)Convergent validity (Correlation to assess it) Divergent validityDivergent validity

03/17/1503/17/15

ValidityValidityVALIDITY

Logical (content)

Criterion related

Congruent (construct)

Face

Ensures adequate and representative set of items that tap the concept

Panel of judges – face validity

Predictive Concurrent

Does measure differentiate to predict a future criterion variable

Analysis – Correlation

Does measure differentiate to predict a criterion variable currently

Analysis – Correlation

Convergent Discriminant

Do the two instruments measuring the concept correlate highly?

Does the measure have low correlation with an unrelated variable?

03/17/1503/17/15

Data Source: SamplingData Source: Sampling

Two Central QuestionsTwo Central Questions

Do we Do we samplesample or or censuscensus? ?

If sample:If sample: How to identify How to identify Who/whatWho/what to include in to include in

the sample? - sampling designthe sample? - sampling design How How manymany to include in the sample? - to include in the sample? -

sample sizesample size

03/17/1503/17/15

What is a Good Sample?What is a Good Sample?

RepresentativeRepresentative of the Population of the Population

Estimates from sample are Estimates from sample are accurateaccurate

Estimates from sample are Estimates from sample are preciseprecise

03/17/1503/17/15

Steps in Sampling DesignSteps in Sampling Design

What is the relevant What is the relevant populationpopulation? ?

What are the What are the parameters parameters of interest?of interest?

What is the What is the sampling framesampling frame??

What What size size sample is needed?sample is needed?

What is the What is the typetype of sample? of sample?

How much will it How much will it costcost??

03/17/1503/17/15

Types of Sampling Types of Sampling DesignDesign

Non-probability

Design

ProbabilityDesign

Convenience

Judgement

Quota

Snowball

Simple Random

Systematic

Stratified

Cluster

Simple Random

Stratified

Combination

Sampling Design

One-stage design

Multistage design

03/17/1503/17/15

Choosing a Sampling Choosing a Sampling DesignDesign

Is REPRESENTATIVENESS critical?

Area samples

Only experts have

information

Info from special interest groups

Quota Judgement

Quick, unreliable

information

Relevant information

about certain groups

Convenience Simple random

Systematic

Cluster if not enough RM

Double samples

Equal sized subgroups?

Proportionate stratified samples

Disproportionate stratified samples

YES NO

Choose PROBABILITY design Choose NON-PROBABILITY design

NOYES

Generalizability

Subgroup Differences

Collect localized

information

Information about

subsets of sample

03/17/1503/17/15

Sample Size: FactorsSample Size: Factors

HomogeneityHomogeneity of sampling units of sampling units

ConfidenceConfidence level level

PrecisionPrecision

Analytical ProcedureAnalytical Procedure

Cost, Time and PersonnelCost, Time and Personnel

03/17/1503/17/15

Roscoe’s Rule of ThumbRoscoe’s Rule of Thumb

Larger than 30 and less than 500 Larger than 30 and less than 500 appropriate for most researchappropriate for most research

A minimum of 30 for each sub samplesA minimum of 30 for each sub samples

Multivariate research: At least 10 times Multivariate research: At least 10 times the number of variablesthe number of variables

Simple Experiments with tight controls Simple Experiments with tight controls - samples as small as 10 to 20- samples as small as 10 to 20