learning outcomes at ijso-2019 - gp globalize
TRANSCRIPT
Page | 1
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Learning outcomes at IJSO-2019
E-ISSN No. 2582-7073
Hala Sultan Saif Al-EasaChemistry and Earth Sciences Department,
Qatar University, P. O. Box 2713, Doha Qatar
AbstractThe 16th International Junior Science Olympiad was held at Doha from 2-12 December 2019. Theanalyses of the test results indicates a strong correlation between the results of MCQ, theory andpractical exam and also a positive correlation in the performance of the students in all the threebranches of sciences (Biology, Chemistry and Physics). The study also discusses the difficultylevel of the test and the quality of the test items based on Discrimination Index calculation.
Keywords: Correlations, Difficulty Index, Discrimination Index, Jr. Science Olympiad
Hala Sultan Saif Al-EasaGPG Journal of Science Education
Vol 1 Issue 1 2020 : pp 1 - 13
1. INTRODUCTION
International Junior Science Olympiad (IJSO) is a sciencecompetition for students under the age of 16, held everyyear in the month of December. The 16th IJSO was heldat Doha, Qatar, by the Ministry of Education and HigherEducation.
Competition was held at Qatar National ConventionCenter from 2-12 December 2019. It consisted of 3separate tests as per the pattern of IJSO: the multiple-choice test (MCQ), Theory part (or the long questions)and the practical part. Practical part is a group eventwhere 3 students from each country form a group toconduct the practical test and each student gets the samemarks for the group event, to be added to their individualMCQ and Theory marks. Each country gets to send 2teams of 3 students each. The host country can sendadditional teams as “guest teams” and hence additionalgroups are allowed.
2. EVENT OF IJSO
The academic event constituted three tests. Multiplechoice test, worth 30 marks was held on the 4th ofDecember 2019, theory questions (long questions), worth30 marks, was held on the 6th of December 2019 and thepractical tests worth 40 marks was held on the 8th ofDecember, 2019. The tasks with their solutions can beseen in reference [1].
The tasks were developed by a team of experts from 18various institutions. The committees consisted of 55number of faculty who were responsible for generatingthe questions. The questions were generated in 23 numberof meetings where each question was discussed at lengthfor the correctness, accuracies, clarity in understandingand deciding if appropriate for the age group underconsideration. All questions covered as many topics ofIJSO syllabus as possible. Table 1 shows the MCQ andtheoretical questions number in relation to the IJSOsyllabus. The practical part, general science skills part
(Received 19 June 2020; Accepted 19 July 2020; Published 20 December 2020)
email: [email protected]
Page | 2
GPG JOURNAL OF SCIENCE EDUCATIONHala Sultan Saif Al-Easa
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
of the syllabus in addition the following criteria were taken into consideration:1. Apply safety regulations2. Acquire basic techniques for measuring3. Observation using all senses4. Identify and use basic laboratory equipment5. Collect data6. Do correct calculations and drawings
Table 1. MCQ and Theoretical Questions in relation to the IJSO Syllabus
Syllabus MCQ Questions Theoretical Questions 1. Particles, waves and matter 4, 9, 12, 13, 23 1, 2, 3, 5 2. Energy 1, 27 2, 3, 4 3. Interactions 2, 14, 15, 16, 17, 19, 20, 22, 24, 25,
26, 28, 30 1, 2, 4, 5
4. Structure, properties and functions 18, 28, 2, 3, 4, 5 5. Systems 3, 5, 8, 10, 11, 21, 29 1, 3, 4, 5 6. Development and Evolution 6, 7 1, 4
Table 2. Distribution of MCQ over the three subjects
Table 3. Distribution of the theoretical questions over the three subjects
Subject Question Number Biology 3, 6, 7, 8, 10, 11, 21, 27, 28, 29 Chemistry 4, 5, 12, 14, 15, 16, 17, 18, 20, 22 Physics 1, 2, 9, 13, 19, 23, 24, 25, 26, 30
Subject Question Number Biology 1 part 3, 2 part 3, 3 part 3, 4 part 3, 5 part 2 Chemistry 1 part 1, 2 part 1, 3 part 1, 4 part 2, 5 part 1 Physics 1 part 2, 2 part 2, 3 part 2, 4 part 1, 5 part 3
The theme of the Olympiad was related to Qatar, but thequestions did not require detailed local knowledge. Whichmeant that the level of questions had to be sufficient toallow differentiation between students in different tests.The main intention was to test not only the students’knowledge, but also their understanding of scientificproblems and their ability to solve them. In addition, theexperimental skills were to be tested in the practicaltests.
The theory questions were tested on students ofbackground in science and with high grade point average(GPA) and practicals were tried out by members of the
Scientific committee for their accuracy of proceduresetc. It is important to highlight that the three scientificdisciplines were integrated into each question, which isclear in the open-ended questions.
In the 16th IJSO, 68 countries participated with team ofstudents (6 each) out of which 13 were guest teams. Thelist of the countries is presented in Table 1. The guestteams are indicated by an *, where these teams were notregular participants but invited for the first time by thehost country.
Page | 3
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 1. Correlations between physics, chemistry, and biology MCQs
3. DATA AND ANALYSIS
A total of 409 students participated in the event (including2 absent) and as per the statues of IJSO [1], 32 studentsfrom 12 countries received gold medals, 64 studentsreceived silver medals and 97 students received bronzemedals. The highest scored student in all guest countrieswere awarded a medal as a sign of appreciation. The
medal distribution of IJSO-2019 is listed in Table 1.
3.1. Multiple choice questionsWhen marks of MCQ were correlated between the threebranches, (Biology, Chemistry and Physics), thecorrelations can be seen in Figure 1, with r value forphysics-chemistry being 0.77, for physics-biology being0.57 and for chemistry-biology being 0.63
Difficulty index for all the 38 questions (30 MCQs, 5 Long questions and 3 practical questions) are plotted inFigure 2 where index of 1.0 is a very easy question answered by every student and 0.0 indicating that no studentanswered it correctly. The difficulty index was calculated using equation (1) which was modified for MCQ withnegative markings as shown in equation (2) where it has a factor of 0.25 x N for a set up when only one optionis correct out of 4 options, and wrong answer is given a negative credit of -0.25. [3]
(r = 0.769)
10
8
6
4
2
0
-2
-4
MC
-Che
mis
try
-4 -2 0 2 4 6 8 10MC-Physics
(r = 0.569)
10
8
6
4
2
0
-2
-4
MC
-Bio
log
y
-4 -2 0 2 4 6 8 10MC-Physics
(r = 0.629)
10
8
6
4
2
0
-2
-4
MC
-Bio
log
y
-4 -2 0 2 4 6 8 10MC-Chemistry
j
j
Total of marks received by all studentsDiff-I = (1)
N × M
Total of marks received by all students+ 0.25 ×NDiff-I = (2)
N× M × 1.25
Page | 4
GPG JOURNAL OF SCIENCE EDUCATIONHala Sultan Saif Al-Easa
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 2. Difficulty Index for all 38 questions
Figure 3. Fraction of correct answer for each question
Figure 3 indicated that more students were able to answer most of the biology questions as compared to other twodisciplines, nevertheless that the difference was relatively small. The distribution of the MCQ marks as shown inthe below histogram (Figure 4) had a wide distribution all over the range of marks thus indicating that the MCQswere neither too easy nor too difficult. Some students received total less than zero in MCQ, because they belongedto some of the guest teams which were participating for the first time and were not familiar with the process ofselection and training students.
1.1
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0 10 20 30 40 50 60 70
Diff
-I
Q. No.
Fra
ctio
n C
orre
ct M
C t
est
0.5
0.4
0.3
0.2
0.1
0.0
SubjectC4
Bio Chem Physsubject
BioChem
Phys
Page | 5
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 4. Histogram of the total score in MCQ
Num
ber
of s
tude
nts
MCQ Marks
50
40
30
20
10
00 5 10 15 20 25 30
3.2. Theory questionsThe 5 theory questions were integrated in nature and each one of them had a physics, a chemistry, and a biologycomponent. Marks for physics from each question were tabled separately and the same was done for chemistry andbiology. A correlation between physics, chemistry and biology marks can be seen in Figure 5.
Phy
sics
Chemistry
10
8
6
4
2
0
0 2 4 6 8 10
Page | 6
GPG JOURNAL OF SCIENCE EDUCATIONHala Sultan Saif Al-Easa
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 5. Correlation between biology, chemistry and physics marks in long questions
Figure 5 shows the correlation between biology-chemistry (r = 0.51), biology-physics (r = 0.54) and chemistry-physics (r = 0.85); where r is Pearson’s coefficients [2]. It also indicated that many students who scored reasonablywell in biology performed quite badly in physics. This is the reason IJSO has been striving hard to integrate thesciences to remove anti-correlation between the three branches of sciences.
Bio
logy
Chemistry
10
8
6
4
2
0
0 2 4 6 8 10
Bio
logy
Physics
10
8
6
4
2
0
0 2 4 6 8 10
Page | 7
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 6. Average marks for each of biology, chemistry and physics for each of the 5 long questions
Figure 6 depicts the average marks received for physics, chemistry, and biology component of each for the longquestion. It is interesting to note that in 4 out of 5 questions, students have scored higher in biology sections andlowest in physics. The situation is however, reversed in question number 5. The detailed analysis of the possiblereasons for such performance is under study and scope of further analysis.
The distribution of the marks received in the long questions is plotted in the below histogram (Figure 7). Thedistribution is skewed towards the lower marks indicating a more challenging set of questions as compared toMultiple choice questions in Figure 4.
Figure 7. Histogram of marks received in the long questions
Num
ber
of s
tude
nts
Theory marks
80
70
60
50
40
30
20
10
00 5 10 15 20 25 30
Q1-Bio
Q1-Phys
Q1-Chem
Q1-Total
Q2-Phys
Q2-Chem
Q2-Bio
Q2-Total
Q3-Phys
Q3-Chem
Q3-Bio
Q3-Total
Q4-Phys
Q4-Chem
Q4-Bio
Q4-Total
Q5-Phys
Q5-Chem
Q5-Bio
Q5-Total
70
60
50
40
30
20
Ave
rage
Sco
re P
erde
nt
Page | 8
GPG JOURNAL OF SCIENCE EDUCATIONHala Sultan Saif Al-Easa
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 8. Correlation between physics, chemistry and biology components of experimental questions
3.3. Practical TestsThe practical Test had 3 questions totaling 40 marks. Each of the 3 tasks had integrated questions and thecorrelation between the physics, chemistry and biology can be seen in Figure 8.
(r = 0.494)
14
12
10
8
6
4
2
0
Pra
ctic
al B
iolo
gy
0 2 4 6 8 10 12 14 16
Practical Chemistry
(r = 0.648)
Pra
ctic
al P
hysi
cs
Practical Chemistry
(r = 0.548)
Pra
ctic
al B
iolo
gy
Practical Physics
14
12
10
8
6
4
2
00 2 4 6 8 10 12 14 16
14
12
10
8
6
4
2
00 2 4 6 8 10 12 14 16
Page | 9
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
The correlation between biology-chemistry (r = 0.49), and biology-physics (r = 0.55) is relatively poor comparedto correlation between chemistry-physics (r = 0.65). The data in Figure 8 represents teams not individual studentsand the work was left for the team to arrange work between them which is not reflected here. The overalldistribution of marks of the practical tests can be seen in Figure 9.
The way experimental questions are designed in IJSO, recording observations in experiments carries certainweightage. Since students are always enthusiastic about carrying out experiments (experience of hands-on activity),it is generally observed that this distribution is skewed towards higher marks and hence the peak of the distributionis not mid-way but more towards 33 marks (out of maximum of 40). Bearing in mind that in such competitionnot only knowledge is measured but also skills and precision.
3.4. Correlation between the three tests formatsCorrelation between different branches of sciences gives an insight on the aspect of interaction. The interactionbetween MCQ, long question and practical can be seen in Figure 10. This correlation is carried out on data whereall the teams attending for the first time are removed, which varies the correlation slightly.
Figure 10. Correlation between MCQ- Long questions and MCQ- practical’s
Num
ber
of s
tude
nts
80
60
40
20
0
Figure 9. Distribution of experimental marks
Practical marks
0 5 10 15 20 25 30 35 40
30
25
20
15
10
5
0
-5 0 5 10 15 20 25 30 35
MCQ
The
ory
30
25
20
15
10
5
0
-5 0 5 10 15 20 25 30 35
MCQ
Pra
ct
Page | 10
GPG JOURNAL OF SCIENCE EDUCATIONHala Sultan Saif Al-Easa
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 11. Correlation between theory questions and practical
The correlation between MCQ and long questions was extremely high with Pearson’s coefficient value of 0.86 forthe correlation between MCQ and theory and the r value was 0.70 for the correlation between MCQ and practical.From Figure 10, it can also be concluded that students scored little higher in MCQ as compared to theory test.Figure 11 shows the correlation between theory and practical where the Pearson’s coefficient was found to be 0.68.
3.5. Discrimination IndexThe Difficulty index and Discrimination Index were calculated for all the 38 questions. These include 38 MCQ,5 theory and 3 Practical tests. The Difficulty index for each of the question was plotted in Figure 2.
The Discrimination Index (DI) for each of the question is plotted in Figure 11.
Figure 12. Discrimination Index values for all 38 questions
40
35
30
25
20
15
10
5
0
Pra
ct
0 5 10 15 20 25 30
Theory
DI
Q No
0 10 20 30 40 50 60 70
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
......Min DI Di......Max DI
Page | 11
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
As per established literature, DI of value > 0.40 is an excellent question,0.30 <DI < 0.39 is a good question,0.20< DI < 0.29 is an average/acceptable question whereas DI < 0.19 is a poor question and should be investigatedfor the sources of low value of DI [2].
The DI values calculated for the present data are given in Figure 11. The DI values were calculated using themethod described in reference [3]. The dotted lines indicate the max and minimum value of DI that is possible foreach of the questions, whereas the average value is indicated with a tolerance level for each question. Themaximum and minimum values of DI result from a subgroup of 38-63 questions.
Figure 13. Summary of correlations between biology, chemistry and biology, in all three forms of test
The DI values are calculated using modified scheme as defined in [3], which handles MCQ with their negativemarking and the long questions scores scaled down to 1 mark. Hence the criteria for deciding the quality ofquestions, based on DI values stated in reference [2], is retained for this work.
4. SUMMARY
From the data in previous sections the overall correlation between physics- chemistry and biology sections of eachof the questions can be summarized in Figure 13. In this event in 2019, the student group presented highercorrelation between chemistry and physics in all the three different forms of competition. However, interestingly,from Figure 14, students scored higher in Biology.
Page | 12
GPG JOURNAL OF SCIENCE EDUCATIONHala Sultan Saif Al-Easa
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
Figure 14. Average grades in all the three different formats of competition
The correlation between the three different formats ofcompetition is particularly good. Students who performedwell in the IJSO 2019 performed equally well in allaspects of questions, thus indicating their preparednessin handling the integrated questions, which is one of themain aims of IJSO.
Out of 38 questions, 30 questions come under “excellent”category, based on the Discrimination Index. FromFigure 2, it can be concluded that this group of studentscould not perform well in 7 out of these 38 questions.To understand and investigate these 7 or 8 questions(with some possible overlap) is the further scope of studyand is under progress.
Similarly, from Figure 2, around this group of studentsperformed very well in about 11 questions. It remains tobe investigated, if these questions were outright simpleor there were other reasons for the group to attemptsolutions to these questions with relative ease.
Figure 8 reflects not only knowledge of individual subjectbut the performance and acquisition of laboratory skills
for the team since not taking into account whichindividual student did the work in a specific subject.
5. ACKNOWLEDGEMENT
Special thanks to Dr. Temadher Khalifa S. Al-Maadeedthe Head of Mathematics, Statistics and PhysicsDepartment, College of Arts and Sciences for directingthe Statistical Consulting UNIT to do the statisticalanalyses and Prof. Ayman Bakleezi for running theanalyses. and. I am also grateful to Dr. P. K. Joshi thepresident of IJSO for his continuous support and guidanceduring the preparation of this paper especially in theparts for correlation between the three test formats andthe Discrimination Index.
I would also like to thank the following institutions forproviding the experts who helped in preparing the testsfor this competition: Ministry of Education and higherEducation, Al Resalah Secondary School, Um HakeemSecondary School, Noor Al Khaleej International Scholl,Al Bayan Secondary School, Qatar Science andTechnology School, Omar Bin Al Khattab Secondary
Page | 13
LEARNING OUTCOMES AT IJSO-2019
GPG Journal of Science Education Volume 1 Issue 1 July-September 2020
School, Mohammed bin Abdulwahab School, Jassim bin Hamad Secondary School, Qatar Secondary School, AliBin Jassim Secondary School, Mohammed Al Mana Secondary School, Omar bin Abdul Aziz Secondary School,Al Shaymaa Secondary School, Doha Secondary School, Qatar University, Weil Cornell University (Qatar) andTexas A& M Qatar University.
6. REFERENCES
1. https://www.ijsoweb.org/downloads
2. Ebel R.L., Frisbie, D.A. Essentials of EducationalMeasurement. 5th edition, Englewood Cliffs, NewJersey: Prentice-Hall Inc; 1991, p232.
3. Joshi P. K., Jain Y., Khunyakari R.K. and Basu S Anovel alternative to analyzing multiple choicequestions via discrimination index (2019).: https://arxiv.org/abs/1906.07941