Using Assessment Program Resources to Shape Effective ELA Instruction

Using Assessment Program Resources to Shape Effective ELA Instruction Amy F. Radikas, Literacy SpecialistRenee A. Savoie, NAEP State Coordinator

Connecticut Assessment Forum IIICrowne Plaza, Cromwell August 13, 2012

1Session OverviewKey shifts in ELA/LiteracyNAEP and PISA basicsEvidence of the shifts illustrated through NAEP & PISADiscussion2CCSS Requires Three Shifts in ELA/Literacy#1Building knowledge through content-rich nonfiction

#2Reading, writing and speaking grounded in evidence from text, both literary and informational

#3Regular practice with complex texts and its academic languageachievethecore.org3ACT ReportIn 2006, ACT released a report called Reading Between the Lines. The findings suggested that the ability to read complex texts is the clearest differentiator between those ready for college-level reading and those not.

4ACT: Reading Between the LinesThe most important implication of this study:

What students could read, in terms of its complexity, was at least as important as what they could do with what they read.

CCSS Appendix A. p. 2

5What is NAEP? National Assessment of Educational ProgressNations Report CardEstablished by Congress in 1969 to answer questions about student academic performanceThe Nations ONLY monitor of what students know and can do common yardstickSurvey designNational, regional and state-level results 6Overall NAEP Reading Results for Connecticut 2003-2011 GRADE 4GRADE 8YEARAVG. SCALE SCORE% OF STUDENTS AT/ABOVE PROFICIENTAVG. SCALE SCORE% OF STUDENTS AT/ABOVE PROFICIENT200322843267*37*200522638264*34*200722741267*37*200922942272* 43X201122742 275X 45X* indicates a statistically significant difference when compared to performance in 2011.7 Grade 4 Reading in 2011: Average Scale Score Comparisons

Higher avg. ss. than CT

Avg. ss. not significantly different from CTLower avg. ss. than CT8Grade 8 Reading in 2011: Average Scale Score Comparisons

Higher avg. ss. than CT

Avg. ss. not significantly different from CTLower avg. ss. than CT Grade 8 Reading in 2011: Average Scale Score Comparisons9Program for International Student Assessment (sponsored by the OECD)

Administered every 3 years since 2000

Assesses 15-year-old students

Measures application of knowledge and skills to problems in a real-life context

What is PISA?

Click to view YouTube video about PISA10PISA aims to answerAre students well prepared to meet the challenges of the future?

Are they able to analyze, reason, and communicate their ideas effectively?

Do they have the capacity to continue learning throughout life?11Who Participates in PISA?

OECD countries (34) Non-OECD education systems (33)12PISA 2009 Reading Literacy ResultsData for 65 education systems (34 OECD and 31 non-OECD)

U.S. not measurably different from OECD average

U.S. scored below 9 education systems

U.S. scored above 39 education systems

13

13PISA Reading Literacy Trends

14PISA and NAEP: Key Differences PISA includes a considerable amount of noncontinuous text.NAEP measures meaning vocabulary.NAEP Grade 8 and 12 passages are notably longer than PISA passages.In terms of readability and grade level, PISA is closer to Grade 12 NAEP.

15The Origins of CCSSInternational AssessmentsNAEPCCSS16CCSS Requires Three Shifts in ELA/Literacy#1Building knowledge through content-rich nonfiction

#2Reading, writing and speaking grounded in evidence from text, both literary and informational

#3Regular practice with complex texts and its academic languageachievethecore.org17Informational TextStudents are required to read very little informational text in elementary and middle school.Informational text is more difficult for students to comprehend than narrative text.Informational text makes up the vast majority of required reading in college/workplace.18NAEP Reading: Passage Types by GradeGradeLiteraryInformational450%50%845%55%1230%70%CCSS ELA p.519In the Early YearsChildren must:Develop strong foundational cognitiveskills (literacy/communication).

20Grade TwoInformational texts use clear and consistent formats, use of simple headings to organize information into categories, illustrations extend the meaning, and simple graphics support understanding of contentElfrieda H. Hiebert, Fundamentals of Literacy Instruction and Assessment, Pre-K6Edited by Martha C. Hougen, Ph.D., & Susan M. Smart

21An Example from NAEPNAEP 2011 Grade 4 Informational Passage: Marians Revolution by Sudipta Bardhan-Quallen Available through the NAEP Questions Tool

22One of 10 Released ItemsWhy is "A Voice for Civil Rights" a good heading for the section that follows it on page 3? Use information from the article to support your answer.23Student Responses: Full Comprehension

24Student Responses: Partial Comprehension

25Student Responses: Little or No Comprehension

26Item-Level Performance DataLittle/NoPartialFullOmittedAvg.SS%Avg.SS%Avg.SS%Avg.SS%NP2054423440245101915CT2073423949249105NP= National Public Reporting standards not met.27CCSS Requires Three Shifts in ELA/Literacy#1Building knowledge through content-rich nonfiction

#2Reading, writing and speaking grounded in evidence from text, both literary and informational

#3Regular practice with complex texts and its academic languageachievethecore.org28Evidence to Support Text Dependent QuestionsCan be literal (checking for understanding) but must also involve analysis, synthesis, evaluation.Focus on word, sentence, and paragraph, as well as larger ideas, themes, or events.Focus on difficult portions of text in order to enhance reading proficiency.Can also include prompts for writing (DOK) and discussion questions.achievethecore.org29Blooms Taxonomy [1956 ] & Blooms Cognitive Process Dimensions [2005]Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recallRemember Retrieve knowledge from long-term memory, recognize, recall, locate, identify Comprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predictApplication -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, writeApply -- Carry out or use a procedure in a given situation; carry out or use /apply to an unfamiliar task Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examineAnalyze -- Break into constituent parts, determine how parts relate Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write Evaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critique Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, valueCreate -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structuresKarin Hess30Webbs Depth-of-Knowledge LevelsDOK-1 Recall & Reproduction - Recall of a fact, term, principle, concept, or perform a routine procedure

DOK-2 - Basic Application of Skills/Concepts - Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems, organize/display data, interpret/use simple graphs

DOK-3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer

DOK-4 - Extended Thinking - An investigation or application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; non-routine manipulations, across disciplines/content areas/multiple sources Karin Hess31The Hess Cognitive Rigor Matrix integrates Bloom + WebbDifferent models used to describe cognitive rigor:

Bloom What type of thinking (verbs) is needed to complete a task?

Webb How deeply do you have to understand the content to successfully interact with it? How complex is the content?Karin Hess32Depth + Thinking Level 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic Thinking/ ReasoningLevel 4Extended ThinkingRemember- Recall, locate basic facts, details, eventsUnderstand- Select appropriate words to use when intended meaning is clearly evident- Specify, explain relationships- summarize identify main ideas- Explain, generalize, or connect ideas using supporting evidence (quote, example)- Explain how concepts or ideas specifically relate to other content domains or conceptsApply- Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning Use context to identify meaning of word- Obtain and interpret information using text features- Use concepts to solve non-routine problems- Devise an approach among many alternatives to research a novel problemAnalyze- Identify whether information is contained in a graph, table, text feature, etc. Compare literary elements, terms, facts, events analyze format, organization, & text structures- Analyze or interpret authors craft (literary devices, viewpoint, or potential bias) to critique a text Analyze multiple sources- Analyze complex/abstract themes

Evaluate Cite evidence and develop a logical argument for conjectures- Evaluate relevancy, accuracy, & completeness of informationCreate- Brainstorm ideas about a topic- Generate conjectures based on observations or prior knowledge- Synthesize information within one source or text- Synthesize information across multiple sources or textsThe Hess Cognitive Rigor Matrix Applies Webbs DOK to Blooms Cognitive Process Dimensions

33DOK is about complexitynot difficulty!The intended student learning outcome determines the DOK level. What mental processing must occur?

While verbs may appear to point to a DOK level, it is what comes after the verb that is the best indicator of the rigor/DOK level.

Describe the process of photosynthesis.

Describe how the two political parties are alike and different.

Describe the most significant effect of WWII on the nations of Europe.Karin Hess34Lets practiceYour class has just read some version of Little Red Riding Hood.

What is a basic comprehension question you might ask?What is a more rigorous question you might ask?

Karin Hess35Depth + Thinking Level 1Recall & ReproductionLevel 2Skills & ConceptsLevel 3Strategic Thinking/ ReasoningLevel 4Extended ThinkingRememberWhat color was Reds cape?Who is this story about?UnderstandApplyIdentify words/phrases that helped you to know the sequence of events in the story.AnalyzeIs this a realistic or fantasy story?Is this a realistic or fantasy story? Justify your interpretation using text evidence.EvaluateCreateWrite a telephone conversation between Red and her mother to explain the wolf incident.

36How can we apply these ideas back in our schools AND CLASSROOMS?

37Guiding QuestionsWhat skills & concepts are most important?

Is the intended rigor of skills/concepts reflected in materials used (e.g., texts, tasks)

Is there a range of DOK (rigor) within the learning activities/lesson?

38Bands11-CCR9-106-84-52-3K-1Increased Ability to Use Text EvidenceStandards Two through Nine Bands11-CCR9-106-84-52-3K-1Standard OneStandard Ten39Increasing Range and Complexityachievethecore.org PISA Reading Unit 3: Graffiti

See Page 1340Question 4: Graffiti (p. 17)Full Credit Guidelines: Explain opinion with reference to style or form of one or both letters. Refers to criteria such as structure of argument, cogency of argument, strategies for persuading readers, etc.

Example: Helgas. She gave you lots of different points to consider and she mentioned the environmental damage that graffiti artists do which I think is very important.

41Question 4: International Performance Data Overall Percent CorrectJapan75>Canada63>United Kingdom62>New Zealand58>Finland58>Australia54=OECD Average53Korea, Republic of52=Ireland51=Poland50=Norway48=United States46