creating valid and effective measures using –optifact- to create better summated rating scales...
TRANSCRIPT
![Page 1: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/1.jpg)
Creating Valid and Effective Measures
Using –optifact- to Create Better Summated Rating Scales
North American Stata Users' Group Meetings
Boston, July 11-12, 2005
![Page 2: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/2.jpg)
Are measures important?
Measures are how theory is translated into data for modeling
Unlike art, where we want to engage individual interpretation as a major component of the work
Measures need to be unambiguously interpreted in precisely the same way by all
![Page 3: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/3.jpg)
Why Scales?
Combining the results of several questions is more reliable and precise, and reduces measurement error (Spector, 1992)
![Page 4: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/4.jpg)
What makes a good scale?
It is reliable• The same measurement, produces similar results across time
and persons
• The questions or items that make up the scale are consistent with one another
It is valid• It measures the concept that it is intended to measure
It measures one concept It is regularly distributed
![Page 5: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/5.jpg)
Reliability
Internal Reliability• Internal consistency: the items should inter-
correlate (Cronbach’s Alpha)
Test-Retest Reliability• Same measure taken again should produce
consistent results
![Page 6: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/6.jpg)
Validity
Rationale• The items must be consistent with theory and the
concept under consideration
Criterion Validity• The items should vary in the same way to other
variables that are theoretically related to the concept
• Canonical correlation
![Page 7: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/7.jpg)
Uni-dimensional, Efficient
Scales that measure a single concept are more useful in establishing causal relationships
Efficient scales use only enough items to reliably measure a concept
Regular distributions allow standard modeling techniques
![Page 8: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/8.jpg)
Research Question
If my scales are not uni-dimensional, are there sub-scales which are and meet the criteria associated with good scales?
Investigated using data gathered for the Canadian National Longitudinal Survey of Children and Youth by Statistics Canada.
![Page 9: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/9.jpg)
Data and Method
The NLSCY cycles 1, 2 and 3• Measures on parenting and child behaviour taken
every two years (allows the investigations of test-retest reliability) N > 13,000.
Common factor analysis• Dimensions, alpha
Canonical correlation• Criterion validity
![Page 10: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/10.jpg)
Problems
There are a lot of candidate scales• 2k – 1, if scales of all sizes are considered
The testing procedure is lengthy• Factor analysis (dimensionality)
• Alpha (internal reliability)
• Canonical Correlation (external validity)
• Re-test (all of the above, twice)
• More than 13 scales
![Page 11: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/11.jpg)
Solution: optifact
A Stata software component to analyze a list of candidate items for the creation of a summated rating scale
![Page 12: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/12.jpg)
-optifact- Specification
optifact varlist [weight] [if exp] [in range]
[, top(#) smallest(#) largest(#) maxfact(#) minslope(#)]
Varlist: the list of candidate items (numeric)
Options:
Top: the number of candidate scales to list (10)
Smallest: the smallest k allowed (3)
Largest: the smallest k allowed (all items)
Maxfact: the maximum number of factors allowed (1)
![Page 13: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/13.jpg)
-optifact- Output
optifact abecq6b abecq6n abecq6w abecq6p abecq6i abecq6qq abecq6hh abecq6s [aweight=awtcw01c], top(20) criteria(childsex ammcq01)
8 items will be processedThere are 219 potential scales 56 combinations of 3 items had one factor, 0 had more than one factor Top 5 scales using 3 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 3 0.774 0.201 Yes abecq6p abecq6i abecq6qq 2 3 0.769 0.203 Yes abecq6p abecq6i abecq6hh 3 3 0.744 0.231 Yes abecq6b abecq6n abecq6hh 4 3 0.742 0.169 Yes abecq6p abecq6qq abecq6hh 5 3 0.734 0.213 Yes abecq6n abecq6p abecq6i
55 combinations of 4 items had one factor, 15 had more than one factorTop 5 scales using 4 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 4 0.802 0.185 Yes abecq6p abecq6i abecq6qq abecq6hh 2 4 0.784 0.199 Yes abecq6n abecq6p abecq6i abecq6hh 3 4 0.776 0.191 Yes abecq6n abecq6p abecq6i abecq6qq 4 4 0.775 0.198 Yes abecq6b abecq6p abecq6i abecq6hh 5 4 0.768 0.172 Yes abecq6p abecq6i abecq6qq abecq6s
![Page 14: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/14.jpg)
Behaviour Scales
Variable Concepts k Alpha
Hyperactivity(becs06) 3 8 0.84
Prosocial Behaviour
(becs07)4 10 0.76
Emotional Disorder /Anxiety (becs07)
3 8 0.78
Aggression
(becs092 6 0.78
Indirect Aggression(becs10) 1 5 0.78
Property Offences
(becs11)2 6 0.62
![Page 15: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/15.jpg)
Parenting Scales
Variable Concepts k Alpha
Positive interaction (0-23 months) (prcs01) 2 5 0.73
Ineffective parenting style (0-23 months) (prcs02)
1 2 0.37
Positive interaction (2-11 years) (prcs03) 2 5 0.81
Ineffective parenting style (2-11 years) (prcs04)
3 7 0.70
Consistency (2-11 years) (prcs05) 2 5 0.68
Rational parenting style (2-11 years) (prcs06) 2 6 0.62
PMK Depression (dpps01)
4 12 0.82
![Page 16: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/16.jpg)
Results
Of these 13 scales only one was uni-dimensional
Most had larger number of items than desirable for the value of alpha
![Page 17: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/17.jpg)
New Behaviour Scales
Variable Concepts k Alpha Concepts’ k’ Alpha’
Hyperactivity(becs06) 3 8 0.84 1 3 0.80
Prosocial Behaviour
(becs07)4 10 0.76 1 4 0.77
Emotional Disorder /Anxiety (becs07)
3 8 0.78 1 3 0.62
Aggression
(becs09)2 6 0.78 1 3 0.67
Indirect Aggression(becs10) 1 5 0.78 1 3 0.70
Property Offences
(becs11)2 6 0.62 1 2 0.63
![Page 18: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/18.jpg)
New Parenting Scales
Variable Concepts k Alpha Concepts’ k’ Alpha’
Positive interaction (0-23 months) (prcs01) 2 5 0.73 1 3 0.63
Ineffective parenting style (0-23 months) (prcs02) 1 2 0.37
Positive interaction (2-11 years) (prcs03) 2 5 0.81 1 3 0.78
Ineffective parenting style (2-11 years) (prcs04) 3 7 0.70
Consistency (2-11 years) (prcs05) 2 5 0.68 1 3 0.66
Rational parenting style (2-11 years) (prcs06) 2 6 0.62
PMK Depression (dpps01) 4 12 0.82 1 3 0.75
![Page 19: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/19.jpg)
Implications
These scales are part of a large survey conducted every 2 years in Canada on more than 20,000 children
The survey has completed 6 cycles of data collection
Each question has been asked more than 120,000 times
![Page 20: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/20.jpg)
Implications (Cont’d)
The scales required 85 questionsThe revised scales require 30 questionsMore than 55,000,000 questions
![Page 21: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/21.jpg)
Caveats
Most scales in the NLSCY are not the best that can be made from the available data
Some of these scales should not be usedThe NLSCY is an excellent survey,
conducted by competent statisticiansSimilar results might be expected in other
major surveys
![Page 22: Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649c8f5503460f94947b71/html5/thumbnails/22.jpg)
Conclusions
-optifact- can help find better measures• Uni-dimensional
• Valid
• Reliable
-optifact- can reduce costs• Equivalent or better measure for same money