Editorial

his last issue of EMYP for 2004 marks the end of my first

about how to encourage and process submissions from col- leagues and about the challenges and anxieties that accom- pany making editorial decisions on behalf of EM:P readers and the NCME community. I have appreciated working with the many researchers who have submitted their work to EM:IP-those whose papers have been accepted and those who did not achieve the results they sought. I look forward to continuing to work with authors and creators of visuals for the covers as we prepare the 2005 and 2006 issues.

I think that you will find this issue ofEM;lP interesting and a bit unique. The two articles address topics that have not re- ceived a great deal of attention in EM:ZP, at least according to my memory and quick perusal of my bookcase. The ITEMS module on standard setting, of course, addresses a prominent topic in these days of standards-based assessment and ex- panding state assessment programs.

Jacqueline Leighton has written a persuasive piece on a now recurring theme in educational testing: incorporating research results and methodologies developed by cognitive psychologists into the work we do on achievement, selection, and other tests in education. She provides what might be thought of as a primer on using think-alouds-also known as cognitive labs-to guide test design and development. She also addresses possible misgivings about the reliability and validity of verbal reports and exhorts us to train psychome- tricians in the basics of cognitive psychology as well as in sta- tistics, research, and measurement.

I consider the article by Andrew Mashburn and Gary Henry particularly unique for EM:IP. I was excited to receive this article bccause it addrcsses assessing young children’s pre- paredness to bcnefit from and succeed in kindergarten (i.e., school readiness). Mashburn and Henry argue that the importance and instances of assessments of young children are expanding and will continue to do so. I believe thatEMYP has not contained work on the assessment of young children, at least in recent memory. This article is unique for EM:IP also because of the analytic methodology. The authors are social policy researchers; their methodology is borrowed from social science studies that investigate systematic bias in the outcomes of social processes. They apply that method- ology-what I would describe as stepwise regression in which they enter blocks of variables rather than individual variables-and use the regression results to identify sources of invalidity in teachers’ ratings of their students’ readiness for kindergarten. I confess that it took me a while to under- stand the analytic logic. Coming from an educational mea- surement point of view, I expected to see the teacher ratings entered as independent variables to examine their accuracy in predicting future success in kindergarten. Instead, in this study, teacher ratings are entered as dependent variables; the blocks of independent variables are used to control and identify biasing influences (e.g., student background) on the teacher ratings. Some analysts may be concerned about the use of ordinary least squares regression rather than mixed or

T year as editor. I have enjoyed this first year of learning hierarchical models. Others may be concerned about the ef- fects of potential multicollinearity within blocks of indepen- dent variables, though the authors would argue that the re- sult would be to miss significant regression coefficients rather than the opposite. (My thanks to Harold Doran for his com- ments on the methodology.) Whatever your point of view may be on the methodology for this study, I think that the authors’ results and recommendations ring true. And, it seems plausible that more and more of our colleagues will have opportunities in the near future to work on assessments for young children. This article introduces some of the issues and challenges they are likely to face.

Finally, this issue contains an ITEMS module. It has been some time since EM;P published a module from theInstruc- tional Topics in Educational Measurement Series. Gregory Cizek, Michael Bunch, and Heather Koons describe widely used standard-setting procedures-specifically, the Book- mark method, recently developed variations on the original Angoff procedure, a.nd the Body of Work method. As the authors point out, it has been about 8 years since the first ITEMS module on standard setting appeared in EM:ZP. They also provide updates on some recent key advances and influ- ences in standard setting, including guidelines €or standard setting in the 1999 Standards for Educational and Psycho- logical Testing and the pervasive influence of standards- based approaches to teaching and learning, test design, and performance standards descriptions.

Cover Visual Dale Cohen of the University of North Carolina, Wilmington and Jon Cohen, my colleague at the American Institutes for Research, have submitted their innovation in graphical dis- plays, a sectioned density plot. As I have written in previous editorials, my goal is to publish visual displays because of the substantive information they provide on a compelling topic or as a means of promoting discussion about methods and de- sign for visual displays. I would add that I am also interested in aesthetic characteristics of visual displays. Here is what Cohen and Cohen have to say about the display.

Presenting whole distributions of several examinee groups for visual comparison poses a difficult challenge. The graphic must present a large amount of information about each group in a limited amount of space on a page. The sectioned density plot (SDP) was designed to engage the reader’s preattentive visual processes-those processes that happen naturally without conscious effort-to increase the amount of infor- mation conveyed.

The SDP makes use of an implied third dimension to display density (or frequency). The illusion of depth is facilitated by combining intensity variation and occlusion. Intensity varia- tion refers to the gradual change of intensity that occurs when light is shined on a three-dimensional object. Occlusion refers to the phenomenon whereby objects closer to the viewer will obscure objects farther from the viewer if the two objects share positions in two-dimensional space. Cognitive psychologists

4 Educational Measurement: Issues and Practice

have demonstrated that intensity variation and occlusion are two of the strongest visual cues to depth that are pre- attentively processed (Cutting 8.1 Vishton, 1995; Ramachan- dra.n, 1988). Whenviewing the SDP, the reader may notice that the sections representing higher density intervals occlude and are brighter than the sections representing lower density intervals.

This example, from the 1999 Grade 8 Trends in Inter- national Mathematics and Science Study (TIMSS), com- pares the mathematics proficiency of US. students across levels of parent education. The three horizontal lines inside the graph mark the boundaries of the 25th, 50th, and 75th percentiles of the combined distribution. The SDP makes simple comparisons a.mong the subgroup distributions im- mediately clear: Students whose parents completed college achieve higher levels of proficiency, and students of less- educated parents achieve lower proficiency scores. More sub- tle comparisons also become simple: The proficiency distri- bution for children of college-educated parents is reasonably symmetrical and quite peaked with somewhat long tails; the distribution in the next category is also symmetric, but broader and with shorter tails; and children of parents who finished primary school show a long lower tail, but a sharp drop-off on the upper side-a pattern that is reversed among children of parents who did not finish primary school. From this graph, one might infer that limited parental education imposes both an upper and a lower bound on children's likely educational attainment-the modal score for students of

"",led Sfale$ Pasta, solV,ce

Statement of Ownership, Management, and Circulation 7 Publlcrimn Tllh 2 P"bl,cat,on Nulnilei

Educational Measurement Issues and Pracl~ce September 24,2004 ~~~

4 lSSLi€ FleqYoncy

Q"aliCMY 540 i $65

Washington. DC 20036 3078

8 Complsm Mailing Address 01 Headwarferi or Gensral BumeG Olllce o/ Publisn~r (Notiorpnnier) or (202) 223-9318 1230 171h S t , NW Washington. DC 20036-3078

- Washington. DC 20036-3078 ..~

last giiartelero12003 Jeri Benson 2004 Steve Ferrara Unwersay 01 Georgia Aderhold Hall, Atlanta. GA 30602.- 1000 Thomas Jelferson St. NW. Washlnyton, DC 20007

~ ~~

Ed,tO, (Name mm#,sie ma,!,"g address)

American Insl,lutes for Research ~~~~~

Managmg Edslor (Name and compiele oiaing afldmsi) Nalional Council on Measurement in EdUCallon 1230 17th St, NW Attention Linda Daobek

~

NallOnal Councll on Measurement 8" Education

Full NmlL

~

even the least-educated parents remains above the lower quartile boundary.

We have programmed the sectioned density plot in AM, a freely distributed statistical software package downloadable at http://am.air.org/.

In Closing As I complete this last issue for 2004, I want to acknowledge the editorial staff at AERA; my colleagues at AIR, Holly Cutting Baker, Rachel Freed, and Kelly Westphalen, who have been so important in managing the review, decision, and editing and publishing processes during this last year; and the superb and almost-always-on-time editorial Advisory Board and outside reviewers for their hard work and wise advice.

References Cutting, J. E., &Vishton, P. M. (1995). Perceiving layout and know-

ing distances: The integration, relative potency, and contextual use of different information about depth. In W. Epstein & S. Rogers (Eds.), Handbook ofperception and cognition: Vol. 5. Perception of space and motion (pp. 69-117). San Diego, CA: Academic Press.

Ramachandran, V. S. (1988). Perception of shape from shading. Nature, 331, 163-166.

Steve Ferrara Editor

Winter 2004 5