Science Curriculum Standards Proficient Level week 9 : Test analysis

Workshop: 9 - 29/11/2011

AGENDA

10:00 – 10:05AM Welcome

10:05 -10: 25 STARTING ACTIVITY

10: 25 - 12:15 test analysis+practice activity

12:15 - 12:30 PRAYER BREAK

12:30- 1:15 Test statistic analysis+practice activity

1:15- 1: 30 FEED BACK

Starting activity

http://www.classtools.net/education-games-php/quiz

Steps to construct your own game

using computer game in revision:

7. Assess and report on students learning.

What do you expect the teachers do to cover this standard?

1- assessment

2- find the result (score).

3. Analyze how well students achieved learning outcomes.

4. Use this information to improve learning

Analyzing Test Questions and Test Scores

TITLE

Let us go to schools and see what analysis we do.

Here are two examples of analysis:

1. Analysis to show whether students achieved the standards.

2. Items analysis.

Discuss the two examples within your group and answer the following question:

Which one would you like to use? If yes explain why?

If No, Give us an alternative and explain why you prefer your analysis?

Academic Year: 2010-2011 Achieving Standards Second Semester

Subject: Chapter 1

school

Class

Student Standard---  standard -----    

standard -----    

standard -----    

standard -----   Standard------ standard

  q----- q------- q---- q---- q---- q---- q---- q---- q---- q---- q----   q---- q---- q---- q---- q---- q--------  

1                                      

2                                      

3                                      

4                                      

5                                      

6                                      

7                                      

8                                      

9                                      

10                                      

11                                      

12                                      

13                                      

14                                      

15                                      

16                                      

17                                      

18                                      

19                                      

20                                      

21                                      

22                                      

23                                      

24                                      

25                                      

26                                      

27                                      

28                                      

29                                      

30                                      

Total                                      

Percentage                                      

Analysis 1

السؤالالمعايير

الرسوب النجاحالعدد

اإلجماليالرسوب نسبة النجاح نسبة

1 7.1 0 40 40 0 100

2 7.3 6 34 40 15 85

3 12.1 4 36 40 10 90

4 12.2 2 38 40 5 95

5 13.1 13 27 40 33 68

6 8.1 6 34 40 15 85

7 8.2 16 24 40 40 60

8 8.3 9 31 40 23 78

9 8.4 9 31 40 23 78

10 11.1 3 37 40 8 93

11 11.2 3 37 40 8 93

12 11.3 6 34 40 15 85

13 quarter 1 1 39 40 3 98

14 quarter 2 3 37 40 8 93

مالحظات المنسق

) اذكر ) سؤال على إلحتوائه الخامس السؤال ثم علل سؤال على ألحتوائه السابع السؤال في النجاح نسب أقلالمعايير ) تحقيق نسبة نقاط أربعة علي اجابته تحتوي والذي % (86األسباب

اإلجراءات يوجد ال

اإلدارة رأي

Analysis 2

12345678910111213140

10

20

30

40

50

60

70

80

90

100

الرسوب نسبة

النجاح نسبة

Generally when analyzing the test items, we have several questions about the performance of each item. Some of these questions include:

· Are the items congruent مالئم with the test objectives?

· Are the items valid? Do they measure what they're supposed to measure?

· Are the items reliableموثوق ? Do they measure consistently?

· How long does it take an examinee to complete each item?

· What items are most difficult to answer correctly?

· What items are easy?

· Are there any poor performing items that need to be discarded تستبعد ?

Item analysis Is process of examining class-wide or course-wide performance on individual test items

Why test analysis?There are several reasons for analyzing questions and tests that students have completed and that have already been graded. Some of the reasons include the following:

1. Identify content that has not been adequately علىكافى ,covered and should be retaught نحو

2 Provide feedback to students,

3. Determine if any items need to be revised in the event they are to be used again or become part of an item file or bank

4.Identify items that may not have functioned as they were intended مطلوب .

5.Direct the teacher's attention to individual student weaknesses

Types of test analysisThere are two types of test analysis: A. Test analysis could be done to ensure that the test developed according to the table of specification.

B. Score test analysis : This includes the following:

1. Item difficulties 2. Distractor analysis for MCQ 3. Test discrimination

3. Test statistic Score tests can be done for all types of questions except test distractor analysis is used only for MCQ.

Score Test Analysis activity: (20 mins)

-Work in group on (one type of test analysis).- Each group read, summarize and add your knowledge.-Each group present their type of test analysis.

Each group should choose:A leader.Presenter.Time keeper.Writer.Observer/s

Note: After the presentation of each type of test analysis, all participants do an activity to self- evaluate her or his understanding

1.Item difficulty analysis: handout 1-a

2.Distractor Analysis: handout 2-a

3. Discrimination Analysis: handout 3-a

1.Item difficulty analysis

•Is the percentage of students who gave the correct answer to each question. (P value)

•It can vary from zero to 100%.

•Item difficulties can be calculated and analyzed for all types of questions.

• See the (Example 1), five true-false questions were part of a larger test administered to a class of 20 students.

•For each question, the number of students answering correctly was determined, then converted to the percentage of students answering correctly

Example 1

Question Correct responses

Item difficulty  

       1     15 75% (15/20)       2     17 85% (17/20)       3 6 30% (6/20)       4     13 65% (13/20)       5     20 100% (20/20)       

The analysis:-Remember that approximately half of the students should answer true-false questions correctly even if they don't know the material because each student has a 50/50 chance of getting the correct answer by guessing.

-Since only 30% of the students answered question 3 correctly in the example, there is an indication of a problem.

- Perhaps the question is not well written and that is misleading,

-Or the material was not covered as thoroughly as the teacher thought. Add more reasons ......

Item Difficulty Practice Activity

Handout 1-b Complete an analyze

2.Distractor Analysis

-In addition to the Item Difficulty index, multiple-choice questions should be examined using a distractor analysis to determine the effectiveness of the various distractors that were provided.

-Example 2:-Shows the responses of 8 students to 5 MCQ.that were part of a longer test.

-The chart shows the frequency with which each response was chosen for each question.

-.

Example 2

 

    Answers to 5 Multiple-choice Questions

     A C A B D <-(correc

t answer

s)  Students Q1 Q2 Q3 Q4 Q5  

  ________________________________________________  

  Ahmad A C A B D  

  Ali A B B B D  

  Jassim A C C D D  

  Amir A B D D D  

  Hamad A C A D D  

  Mohamad B C B B A  

  Mahr C C C D A  

  Abd al rhman

D C D D B

A B C D

  ________________________________________________

  Q#1 5** 1 1 1

                

  Q#2 0 2 6** 0

                

  Q#3 2** 2 2 2

                

  Q#4 0 3** 0 5

                

  Q#5 2 1 0 5**

** Denotes Correct Answer

- Over 50% of the students answered question #1 correctly, and each of the distractors was selected.

-The distractors have functioned as they should.

   It is not desirable to have one of the distractors chosen more often than the correct answer, as occurred with question 4.

This result indicates a problem with the question. Distractor D may be too similar to the correct answer and/or there may be something in either the stem or the alternatives that is misleading.

        

If students do not know the correct answer and are purely guessing, their answers would be expected to be distributed among the distractors as well as the correct answer, much like question 3.

If one or more distractors are not chosen, as occurs in questions 2, 4, and 5, the unselected distractors probably are not plausible.

If the teacher wants to make the test more difficult, those distractors should be replaced in subsequent tests.

Distractor Analysis Practice Activity

Handout 2-b Complete an analyze

3. Discrimination Analysis

Computing the item discrimination index enables you to determine whether the question discriminates appropriately between lower scoring and higher scoring students.

When students who earn high scores are compared with those who earn low scores, we would expect to find more students in the high scoring group answering a question correctly than students from the low scoring group.

•What we would not expect to find is a case in which the low scoring students answered correctly more frequently than students in the high group.

•The item discrimination index (labeled "D" in Example 3) can vary from -1.00 to +1.00

•A negative discrimination index (between -1.00 and zero) results when more students in the low group answered correctly than students in the high group.

•A discrimination index of zero means equal numbers of high and low students answered correctly, so the item did not discriminate between groups.

• A +ve index occurs when more students in the high group answer correctly than the low group. Students who are fairly homogeneous in ability and achievement, their test performance is also likely to be similar, resulting in little discrimination between high and low groups.

• Questions that have an item difficulty index (NOT item discrimination) of 1.00 or 0.00 need not be included when calculating item discrimination indices.

•An item difficulty of 1.00 indicates that everyone answered correctly, while 0.00 means no one answered correctly.

• procedure in analyzing a test for item discrimination (see h.o.3-a)

Example 3

 

  Number in high group answering correctly

Number in low group

answering correctly

  H - L dividedby the number

in each group (8)

Question

   H       L    H - L    D   

________________________________________________________________

1 8 4 4 .5         2 7 2 5 .625         3 5 6 -1 -1.25

-For question 2, only 7 in the high group answered correctly, 2 in the low group.

-Dividing the difference of 5 by the number in the groups (8) results in .625.

- Both questions one and two provide results in the expected direction. We would expect more of the students with high scores to answer correctly than students with low scores.

- When this situation occurs we have a positive index.

- Question 3, however, shows an unexpected result, with more of the low scorers getting the question right than high scorers. Note that when we subtract L from H in this case the result is a negative number, and therfore a negative index.

-This is a clue that there may be a problem with the way the question was presented on the test or the way the material was taught (or not taught).

Discrimination Analysis Practice Activity

Handout 3-b Complete an analyze

Prayer Break

4.Test statistic analysis:

This include the following1.The mean – الوسط الحسابي المتوسط الحسابي

The mean of a test is the arithmetic mean, calculated by adding all the scores on the test, then dividing by the number of test takers.

-In example 4 a frequency distribution has been prepared by listing the scores in order from low to high.

- The sum of the scores for the 10 students who took the test is calculated (453).

This is divided by the number of students (10) to find the mean is 45.3. Example 4

Scores 

50 

48 

48 

47 

45 

45 

45 

43 

42 

40

In some circumstances, the mean may not be the best indicator of student performance. If there are one or a few students who score considerably lower (or higher) than the other students, their scores tend to pull the mean in their direction.

In this case the 2. median الوسيط is usually considered a better indicator of student performance.

To determine the median of a group of test scores, first list the scores vertically from highest down to lowest. If there is an even number of students, the median is the score that divides the scores into two equal groups.

In example 4, our line to divide the group into two equal halves would be between the two scores of 45, thus our median would be 45

 In example 5, our line would be between 44 and 45, so the median would be halfway between them at 44.5. In this case the median is not an actual score earned by one of the students. In example 6, the distance between the two middle scores (43 and 46) is more than one, so we again find the point halfway between them for our median of 44.5.

If the number of students is uneven, determining the median is less complicated. In that case, find the one score that is the middle score in the frequency distribution, having equal numbers of scores above and below it. In the scores below, the median is 44 in example 7, and 45 in example 8. It does not matter if more than one student earns that score, as in example 8.

Example 5 Example 6 Example 7 Example 8

  Scores Scores Scores Scores

  50 50 49 50

  48 49 48 49

  48 48 48 47

  47 46 47 47

  45 46 45 45

  44 43 44 45

  43 43 43 45

  42 42 42 44

  42 41 42 42

  41 41 41 41

      38 41

The 3.MODE المنوال is the score (or scores) that occur most frequently. In example 4 (above) there are 3 scores of 45, so that would be the mode for that test. In example 5 there are 2 scores of 48 and 2 scores of 42 so there are two modes. When there are two modes, this is called a bimodal distribution.

The same situation occurs in examples 6 and 7, but not in example 8.    

The 4. RANGE المدى describes the variability in the scores by showing the lowest and highest scores earned. In example 4, the range would be 40-50, in examples 5, 6, and 8, 41-50, and example 7, 38-49. Statisticians apply the term range in a slightly different manner but still use it to determine the spread in scores.

Citing the lowest and highest scores provides additional information, indicating whether the range of scores was at the high or low end of the scores (or in between), not just how far apart the lowest and highest scores.

extention5. Standard deviation: range above and below the average score, the more the scores are spread out the high the SD.

Test Statistic Analysis Practice Activity(+discrimination)

Handout 4-b Complete an analyze

How to Use Information gained from analysis

1. Identify strengths in student learning

- continue doing what you’re doing!

2. Use identified weaknesses to improvestudent learning by:

- Providing feedback to correct misconceptions

- Modifying course content

- Changing your approach for teaching aconcept or skill

REFLECTION

Feedback