vce data service users manual version 5 - weebly...gat will be influenced by what the school has...

V I C T O R I A N C U R R I C U L U MA N D A S S E S S M E N T A U T H O R I T Y

VCE Data Service Users Manual

Version 5

Victorian Curriculum and Assessment Authority

2014

© Victorian Curriculum and Assessment Authority (2014)

No part of this publication may be reproduced except as specified under the Copyright Act 1968 or by permission from

the VCAA. For more information go to: www.vcaa.vic.edu.au/Pages/aboutus/policies/policy-copyright.aspx

The VCAA provides the only official, up-to-date versions of VCAA publications. Details of updates can be found on the

VCAA website: www.vcaa.vic.edu.au

This publication may contain copyright material belonging to a third-party. Every effort has been made to contact all

copyright owners. If you believe that material in this publication is an infringement of your copyright please email the

Copyright Officer: [email protected]

VCE Data Service User Manual iii

Contents

General information 1

Overview 1

Section 1 – VCE Data Service report structure 5

Section 2 – A strategy for using the VCE Data Service 9

How were the school’s overall results this year? 9

Report 1 9

Report 2 9

Report 3 9

Report 4 9

Report 5 9

Report 6 9

What was the level of results like in each study? 10

Report 7 10

Report 8 10

Report 9 10

How did our results compare to reasonable expectations? 10

Report 10 10

Report 11 10

Report 12 10

Report 13 10

Report 14 10

Is our student cohort changing over time? 11

Report 15 11

Report 16 11

What can we learn about group patterns from the performance of individual students? 11

Report 17 11

Report 18 11

Section 3 – Interpreting and using adjusted scores 13

iv VCE Data Service User Manual

Section 4 – VCE Data Service reports 15

Combined Study Reports 15

Report 1: All VCE Studies 15

Report 2: Assessment Grades 16

Report 3: Study Score Frequencies 17

Report 4: Completions 18

Report 5: Study Scores of 40 and above 18

Report 6: Region School Groupings 19

Single Study Reports 20

Report 7: Selected VCE Studies (Study Scores) 20

Report 8: Single VCE Study (Study Scores and Graded Assessments) 21

Report 9: Single VCE Study (Study Scores x Time) 22

Adjusted Score Reports 24

Report 10: Selected VCE Studies (Adjusted) 24

Report 11: Single VCE Studies (Adjusted) 25

Report 13: Subgroup Comparison (Adjusted) 28

Report 14: Achieved versus Predicted Scores 29

GAT Score Reports 30

Report 15: GAT and Study Score Distributions 30

Report 16: GAT and Study Score Distributions x Time 31

Student-level Reports 32

Report 17: Student Results by Study 32

Report 18: Student Results across Studies 33

Appendix 1: Glossary of terms 35

Appendix 2: Generating VCEDS reports 49

Report 1: All VCE Studies 49

Report 2: Assessment Grades 50

Report 3: Study Score Frequencies 51

Report 4: Completions 52

Report 5: Study Scores of 40 and above 53

Report 6: Region Schools Grouping 54

Report 7: Selected VCE Studies (Study Scores) 55

Report 8: Single VCE Study (Study Scores and Graded Assessments) 56

Report 9: Single VCE Study (Study Scores x Time) 57

Report 10: Selected VCE Studies (Adjusted) 58

Report 11: Single VCE Study (Adjusted) 59

Report 12: Single VCE Study (Adjusted) x Time 60

Report 13: Subgroup Comparison (Adjusted) 61

Report 14: Achieved versus Predicted Scores 62

Report 15: GAT and Study Score Distributions 63

Report 16: GAT and Study Score Distributions x Time 64

Report 17: Student Results by Study 65

Report 18: Student Results across Studies 66

Appendix 3: Report short-cut guide 67

General Information VCE Data Service User Manual 1

General information

OverviewEach year, a massive amount of information on the abilities and achievements of students is collected as part of the annual Victorian Certificate of Education (VCE) assessment program. As schools have become more sophisticated in their efforts to improve outcomes for their students, they have become increasingly interested in accessing performance data.

Over the years, the Victorian Curriculum and Assessment Authority (VCAA), and its predecessor organisations, have conducted trials into various ways of summarising and reporting information in both hard copy and electronic form. The VCE Data Service (VCEDS) is the outcome of many years of experience in providing schools with confidential information on the performance of their VCE students.

The VCEDS was first launched in February 2003 and since then there have been a number of enhancements.

Schools can access information for their school and compare the results with aggregated results for all schools and schools within the same sector. Users can choose from a wide array of options for displaying VCE results.

Schools can choose whether to focus on a particular year or on results for several years.

Schools can examine results for a single VCE study, for various combinations of studies (including by Study Groups), or for all studies offered within the school.

Schools can choose whether to look at VCAA Study Scores, Australian Tertiary Admissions Rank (ATAR) Subject Scores, or assessment grades awarded on each graded assessment. If schools choose to look at Study Scores, they may wish to examine the scores that their students have achieved, or how well they performed compared to what might have been expected given their gender and ability. For an individual, this involves the calculation of an adjusted score; for a class, the aggregation of adjusted scores is referred to as an adjusted estimate.

Adjusted scores are the difference between how a student (or group) achieves in terms of a study score compared to a predicted study score. The predicted study score, calculated by the VCAA, incorporates a student’s ability (as estimated by the General Achievement Test), a student’s gender and the ability of the school cohort in the study (which is also estimated by the General Achievement Test). As a result, an adjusted score is simply a measure of whether a student (or group) has achieved a result that was:

• higher than predicted

• as predicted

• lower than predicted.

As a result, when interpreting adjusted scores it is vital to consider what is and what is not incorporated in the prediction. There are a range of other factors the predictions do not directly take into account, such as student attitude, student background, teacher performance, school resourcing and significant events (for example, student sickness). As a result, the adjusted scores are a combined measure of these localised issues, which the prediction does not incorporate. The challenge for schools is to identify which factors have influenced their students outcomes.

Adjusted estimates cannot be regarded as definitive, and never explain why students perform better or worse than predicted. This can only be addressed at an individual school level and in the light of local knowledge about the school, and its students and staff.

2 VCE Data Service User Manual General Information

All adjusted estimates are associated with a degree of uncertainty or error. To indicate this degree of uncertainty, adjusted estimates are accompanied by confidence intervals and coefficients indicating the reliability of adjusted estimates.

Menus allow schools maximum flexibility in deciding what kinds of reports they wish to view on the screen and in deciding whether to print or copy selected reports for pasting into other documents.

There is an online help manual that provides an explanation of the various reports and also of the key terms and concepts involved and suggestions for using the information.

The purpose of the VCEDS is to provide schools with information that will be of assistance to them as they seek to evaluate the quality of education provided to their VCE students. It is intended as a tool to be used for improvement purposes, and not as a report card on the school. The use and interpretation of VCEDS reports requires careful reflection, detailed local knowledge and a commitment to systematic, ongoing improvement.

Adjusted scores

VCE results tell us a lot about the performance of students in VCE studies relative to all other students taking those studies. They are not, however, particularly good indicators of the extent to which schools have contributed to the levels of attainment of their students. When comparisons are made of the average performance of students attending different schools, what is revealed often indicates more about the nature of the students attending those schools than it does about the effectiveness of the instruction provided by schools. Similarly, within schools, it is difficult to say whether the performance of students is a reflection of the quality of the teaching provided or of the abilities of the students enrolled in a given study.

The VCEDS makes use of sophisticated multilevel regression modelling to generate adjusted scores. These are scores that have been adjusted for the effects of two of the most powerful predictors of student performance, namely student academic ability and gender.

Let us first consider adjustments for ability. Midway through their final year, all VCE students sit the GAT. The GAT provides measures of the general knowledge and skills that students have built up during their school years in three areas:

• Written Communication

• Mathematics, Science and Technology

• Arts, Humanities and Social Science.

These skills are built up over a period of time prior to Year 12 and provide an indication of how well students are equipped to undertake VCE studies. For this reason, it is a good predictor of VCE performance and can be used as an indicator of where students would be in their VCE studies, ‘other things being equal’. The GAT is therefore regarded as a measure of general ability or preparedness for the VCE.

Given both an achievement score and a set of measures of general ability, it is possible to estimate the effect of the school on student achievement, having statistically adjusted for the abilities of the students in each school. This is not an ideal adjustment because it makes use of an ability measure obtained midway through the students’ final year of secondary schooling. It is reasonable to assume that student performance on the GAT will be influenced by what the school has already contributed to students’ general academic abilities to that point in time. As such, the GAT is not an entirely appropriate measure to use to adjust scores if the purpose is to compare one school with another. However, it is a very appropriate adjustment when used to assess the relative effectiveness of Year 12 teaching in different studies within a given school.

In the reports provided as part of the VCEDS, adjustments are made for individual student ability as represented by their scores on all three sub-tests of the GAT, and also for the overall ability of the students taking a given study within the school. It is a well known fact that students’ performances are affected by their abilities, and also by the abilities of those around them. Two students of the same ability will perform differently if placed in classes comprising students of different overall levels of ability. The student placed in the higher ability class is likely to perform at a higher level than the student placed in the lower ability class.

General Information VCE Data Service User Manual 3

For this reason, the average total GAT score of the students taking a given study is computed and taken as a measure of the overall ability of the student group. Each student’s actual score is adjusted not just for the student’s individual ability but also for the average ability of all students taking that study within the school.

The other variable used to adjust actual scores is gender. For many years there has been evidence of significant differences in the performance of male and female students in VCE studies, and indeed in measures of student performance at all levels of education and in almost all education systems. These differences are quite substantial, particularly in written communication, with girls out-performing boys. Given the fact that the proportion of females to males varies significantly from school to school and from study to study, it is appropriate to adjust for the effects of gender, since any assessment of the effectiveness of the school or of the quality of the teaching could be confounded by the influence of this factor.

A frequently asked question is whether adjusting for ability and gender really enables fair comparisons to be made between schools and between studies within schools. Should there not also be adjustments for the socioeconomic characteristics of students? The answer to this question is that adjusting for student academic ability in large measure captures the impact of home background factors, since academic ability measures themselves reflect the effects of unmeasured home and background characteristics. In addition, collecting accurate and comprehensive socioeconomic and other family and individual background information raises serious issues of intrusion into peoples’ private lives, as well as often insurmountable logistical problems of collecting the required information from all students.

Finally, let us turn to consider the adjusted scores themselves. Like the VCE Study Scores, adjusted scores have been calculated using the same scale as Study Scores.

A predicted score is calculated for each student, based on their gender, their three GAT scores, and the average GAT score of students in the school undertaking the particular study.

The predicted score is subtracted from the score they actually achieved. The result, an adjusted score is an indication of how well each student performed compared to what might have been expected given the performance of comparable students across the state.

The adjusted scores are aggregated over a student group. The result, an adjusted estimate, is a measure of how well the student group performed compared to expectations.

If students perform precisely as predicted in a given study, taking into account their abilities and their gender composition, then the school will have an adjusted estimate of zero. If the students perform less well than predicted, the school will have a negative adjusted estimate (for example, if the adjusted estimate is -2.5, this means that students performed 2.5 Study Score points below what was predicted) and if students perform better than predicted, the school will have a positive adjusted estimate (for example, if the adjusted estimate is +1.7, this means that students performed 1.7 Study Score points above what was predicted).

The adjusted estimates are always associated with a degree of uncertainty or measurement error. They are thus accompanied by confidence intervals and coefficients indicating the overall reliability of the adjusted estimates.

To minimise problems of unreliability of results due to small numbers of students enrolled in a study in a particular school or small numbers of schools enrolling students in a particular study, the following rules were adopted in calculating adjusted estimates. First, for a given study, all schools with less than five students with complete data were excluded from the analysis. Second, no adjusted estimates were calculated for any study in which there were less than 20 schools and 400 students.

VCE Data Service reporting structure VCE Data Service User Manual 5

Section 1 – VCE Data Service report structure

SummaryUpon entry to the VCEDS system, users are presented with a menu (Figure 1.1) that invites them to choose from 18 reports in five categories: Combined Study Reports 1–6, Single Study Reports 7–9, Adjusted Score Reports 10–14, GAT Score Reports 15–16 and Student-level Reports 17–18. Reports have been numbered for ease of reference.

The report structure has been designed to reflect the way that many of us approach data – first it gives a very broad overview of across-the-board performance. Then it moves to a closer level of detail in performance on individual studies, and on the three assessment tasks in each study. Then it zeroes in on performance compared to expectations, allowing the user to make all kinds of comparisons of the performance of subgroups of students within a study. Finally, it allows the user who wants to understand the results at a more intimate level to examine the contribution of individual students to the pattern of school or class results.

As you become more familiar with VCEDS, you will develop your own approach, and will almost certainly skip several of these steps. Until you achieve this familiarity, the sequence of numbering of the reports in VCEDS may provide you with a useful guide.

Figure 1.1

6 VCE Data Service User Manual VCE Data Service reporting structure

The first set of reports (Report 1 to Report 6; see Figure 1.2) looks at summary results across sets of studies combined, and will be of particular interest to school administrators.

Figure 1.2

The second (Report 7 to Report 9; see Figure 1.3) looks at summary results in specific VCE studies and may be of interest both to school administrators and subject coordinators.

Figure 1.3

The third category (Report 10 to Report 14; see Figure 1.4) provides reports of adjusted estimates for specific VCE studies and may be of interest both to school administrators, subject coordinators and teachers.

Figure 1.4

VCE Data Service reporting structure VCE Data Service User Manual 7

The fourth set of reports (Report 15 and Report 16; see Figure 1.5) provides details of GAT scores and enables schools to track changes in the ability of their student cohort from year to year, and to monitor the relationship between these changes and changes in student achievement.

Figure 1.5

The final category of reports (Report 17 and Report 18; see Figure 1.6) provides reports at the individual student-level. These reports allow schools to better understand the patterns of class results by identifying particular students or groups of students who may have exceeded or fallen short of expectations.

Figure 1.6

The Combined Study Reports 1–6, Single Study Reports 7–9, Adjusted Score Reports 10–14, GAT Score Reports 15–16 and Student-level Reports 17–18 are detailed in the VCE Data Service reports section (see page 15).

A strategy for using the VCE Data Service VCE Data Service User Manual 9

Section 2 – A strategy for using the VCE Data Service

Because the volume of information that could be extracted from the VCEDS is so great, it is important that users approach it with a strategy in mind. Browsing around can be fun, but it is unlikely to lead to systematic information that will generate school improvement.

You may wish to approach the service by asking the following questions.

How were the school’s overall results this year?

Go to the Combined Study Reports first. Look at the following reports:

Report 1

What was the overall level of performance? How does it compare with other schools in general? How does it compare with the results achieved in your school last year, and in previous years?

Report 2

What is the overall distribution of grades last year? Did you have the number of high grades, or low grades that you expected? Were there more NAs and UGs than you would expect? If so, are they concentrated in some Study Groups (SGs) or are they spread evenly across the curriculum?

Report 3

You might, out of curiosity, like to examine just how many scores of 50, 49, 48, etc. were achieved in a given year. This may be useful if your school focuses on particular targets, particularly with very high Study Scores.

Report 4

Are students completing the basic requirements satisfactorily? Are you getting students through, even those who are not high flyers? Are your results improving in this respect, compared to previous years?

Report 5

Are you producing the number of very high achievers that you would like to? Are there more or less of them than in previous years?

Report 6

Which schools are in the same sector and region as my school? This information is useful when comparing your school’s result with other schools in the same sector and region.

10 VCE Data Service User Manual A strategy for using the VCE Data Service

What was the level of results like in each study?

Look next at the Single Study Reports. These will show you the overall level of results in each study, in terms of scaled or unscaled Study Scores (ATAR Subject Scores and VCAA Study Scores), as well as in terms of the grades achieved on each assessment.

Report 7

This allows you to look at results across studies, and to identify studies that you may want to look at more closely.

Report 8

This report provides you with a detailed look at results in a single study, including performance on each graded assessment. You may look at the performance of different groups of students in the study, including gender groups, class groupings, Year 11 and Year 12 students, and other groupings that you may wish to create within your own school.

Report 9

This report allows you to look back at the pattern of results in any study over as many years as you choose. Are this year’s results part of an encouraging trend, part of a worrying trend, or a ‘blip’ on the radar screen? If special efforts have been made in particular studies, have they led to the desired improvement?

How did our results compare to reasonable expectations?

Adjusted estimates can answer this question for you, so look next to the Adjusted Score Reports.

Report 10

This allows you to look at adjusted estimates across studies, and to identify studies in which the results have been significantly higher or lower than you might have expected given the student cohorts in those studies.

Report 11

This report provides you with a detailed look at the adjusted estimates in a single study, and the ability to compare these to the performance of students in other schools. This helps you to view your own students’ achievements in the wider context of other students in other schools across the state.

Report 12

This report allows you to look back at the pattern of adjusted estimates results in any study over several years. If there is a change in the pattern of results, is it explainable in terms of changes in the student cohort, or is there something else happening?

Report 13

This report provides you with a detailed look at the adjusted estimates in a single study. You may look at the performance of different groups of students in the study, adjusted for gender and ability, including class groupings, Year 11 and Year 12 students, and others that you create within your own school. Maybe you can identify categories of students that are reaching expectations, and categories of students that are not.

Report 14

What was the pattern of results that you might have expected in each study, and what was the pattern of results that your students actually achieved?

A strategy for using the VCE Data Service VCE Data Service User Manual 11

Is our student cohort changing over time?

You may suspect that your student cohort is significantly more able, or significantly less able this year than in previous years, or you may suspect that there is a long-term trend at work. You may suspect that changing patterns of results are associated with changing patterns of student ability. These reports allow you to ask and answer questions about these issues.

Report 15

What was the pattern of ability of students in your school, or in any given study, as indicated by the three GAT component scores and the GAT total score. How do the patterns of Study Scores and ATAR Subject Scores compare with these?

Report 16

Is there a trend in the overall pattern of GAT results in your school over time? Are there trends in individual studies? And, importantly, if there are such patterns, do the patterns of results that your students have achieved mirror or defy these trends?

What can we learn about group patterns from the performance of individual students?

Are there individuals whose results surprised you, and what can be learned from the study of individual cases? Where individuals have excelled, did they excel across the board, or was it confined to a single study (perhaps the one that you teach)?

Report 17

Look at the pattern of achieved and expected scores in each study individually. Where do particular groups of students appear in that pattern? Who were the exceptions, and how well do you understand the reasons for their exceptionality?

Report 18

Where an individual student has excelled in comparison with expectations, was it across the board, or was it just in a single study? Similarly, where a student fell short of expectations, was it consistent, or confined to one or two studies? What can we learn from the students who have consistently exceeded our expectations, and what can we learn from our knowledge of those who consistently fell short?

Interpreting and using adjusted scores VCE Data Service User Manual 13

Section 3 – Interpreting and using adjusted scores

In interpreting adjusted estimates, the following features should be noted:

1. An adjusted score is the difference between the score actually achieved by a student and the score predicted on the basis of gender and GAT performance. These are aggregated into adjusted estimates that describe the performance of a group of students in a school compared to expectations.

2. All adjusted estimates are centred on zero. An adjusted estimate of zero does not imply that the students have made no progress, but rather that their progress has been average. A positive adjusted estimate means that the students have performed better than predicted given their ability and gender profile, while a negative adjusted estimate means that the students have performed less well than predicted.

3. Adjusted estimates are reported on the same scale as Study Scores. If students perform precisely as predicted in a given study, taking into account their abilities and their gender composition, the school will have an estimate of zero. If the students perform less well than predicted, the school will have a negative estimate (for example, if the estimate is -2.5, this means that students performed 2.5 Study Score points below what was predicted) and if students perform better than predicted, the school will have a positive estimate (for example, if the estimate is +1.7, this means that students performed 1.7 Study Score points above what was predicted).

4. The adjusted estimates are what are known as ‘empirical Bayes estimates’, which means that they are weighted to reflect the reliability of each school’s estimate. As a consequence, the adjusted estimates for schools with small numbers of students enrolled in a particular study are biased towards the overall average. For very small schools in which students perform much better than predicted, their true performance will not be reflected fully in their adjusted estimates, which will be ‘shrunk’ towards the average (zero). The same applies in the opposite direction to very small schools in which students perform much worse than predicted. The lack of performance of the students will not be reflected in the school’s adjusted estimates. In other words, the estimates are conservative estimates.

5. Schools with less than five students with complete data in a given study are automatically excluded from analyses to generate ability and gender adjusted scores. This is because adjusted estimates based on very small numbers of students are inherently unreliable.

Adjusted estimates cannot be regarded as definitive, and never explain why students perform better or worse than predicted. This can only be addressed at the individual school level and in the light of local knowledge about the school, and its students and staff.

In interpreting results, it is important to look for trends over time and for significant differences. Statistically significant differences are evident when there are non-overlapping confidence intervals. Notice should be taken of consistently high or low performance over two or more years, but caution should be exercised in reading too much into what may simply be random fluctuations from one year to the next.

A number of factors may explain higher or lower than predicted performance and it is important to take time to understand properly the pattern of results. For example, some schools have found that in some studies student performance is consistently lower than predicted despite good teaching. On further investigation they have concluded that this is because their students have not taken these studies as seriously as they have their other studies, which they perceive as more critical to meeting tertiary entrance criteria.

In most cases, schools will find the greatest benefit in focusing on the handful of studies in which students appear to be performing significantly higher or lower than expected.

VCE Data Service reports VCE Data Service User Manual 15

Section 4 – VCE Data Service reports

Combined Study ReportsThe VCEDS provides the following reports of summary results across sets of studies combined and school comparison report:

• all VCE studies

• assessment grades

• study score frequencies

• completions

• study scores of 40 and above

• Region Schools Grouping.

Report 1: All VCE Studies

The All VCE Studies report (Figure 4.1) displays the distribution of VCAA Study Scores or ATAR Subject Scores for all VCE studies in a particular year for This School and allows for comparison with various School Groups. For an explanation of School Groups, please refer to Appendix 1 – Glossary of Terms.

The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A. A gender and year level breakdown of each distribution is optional.

The distribution of scores is indicated by means of box-and-whisker plots in which the box encloses the middle 50% of schools, the top of the box represents the 75th percentile, the bottom of the box the 25th percentile, the top of the whisker the 90th percentile, the bottom of the whisker the 10th percentile. The heavy line in the middle of the box indicates the 50th percentile. A legend to the left of each chart serves to remind users of this.

Figure 4.1

16 VCE Data Service User Manual VCE Data Service reports

The table in Figure 4.2 summarises the distribution of all Study Scores in a particular year for All Schools, schools in This School Sector and This School.

Figure 4.2

Report 2: Assessment Grades

The Assessment Grades report (Figure 4.3) displays the percentage of assessment grades for all VCE studies in a particular year for This School. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their Assessing school.

Users can choose whether they want comparisons with the various School Groups.

Figure 4.3

The table in Figure 4.4 summarises the total number of assessments with grades A+ to UG and A+ to UG plus NA for the various School Groups and This School for a particular year.

Figure 4.4


The table in Figure 4.5 summarises the percentage of assessments at each grade level of all assessments for the various School Groups and This School for a particular year. The percentage of assessments at each grade level is expressed as a percentage of all assessments (excluding NA) while the percentage of NA assessments is expressed as a percentage of all assessments including NA.

Figure 4.5

Report 3: Study Score Frequencies

The Study Score Frequencies report displays the distribution of VCAA Study Scores for all VCE studies in a particular year for the various School Groups and This School. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their Assessing school.

The table in Figure 4.6 summarises the total number of assessed studies (schools by the number of studies offered with assessment results of A+ to UG) and the total number of assessed and NA studies (schools by the number of studies offered with assessment results of A+ to UG or NA) for the various School Groups and This School for a particular year.

Figure 4.6

The table in Figure 4.7 summarises the percentage of assessments at each study score level for the various School Groups and This School for a particular year. Note that this report is best printed in landscape format.

Figure 4.7


Report 4: Completions

The Completions report (Figure 4.8) lists the number and percentage of students in the school completing each of Units 3 and 4, and the number who completed the requirements for the award of a Study Score in a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 4.8

Report 5: Study Scores of 40 and above

The Study Scores of 40 and above (see Figure 4.9) report displays the percentage of Study Scores of 40 or above for all VCE studies for selected years for the various School Groups and This School. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school.

Figure 4.9


The table in Figure 4.10 summarises the percentages of Study Scores of 40 or above for selected years for All Schools, schools in This School Sector and This School.

Figure 4.10

Report 6: Region School Groupings

Some of the reports provide an option that allows the results of This School to be compared with schools in This School Region. The Region Schools Grouping report provides a list of schools that belong to the same sector and region as This School. This report only applies to Catholic and government schools. For a more detailed explanation, refer to Appendix 1 – Glossary of Terms.


Single Study ReportsThe VCEDS provides the following reports of summary results in specific VCE studies:

• Selected VCE Studies (Study Scores)

• Single VCE Study (Study Scores and Graded Assessments)

• Single VCE Study (Study Scores x Time)

Report 7: Selected VCE Studies (Study Scores)

The Selected VCE Studies (Study Score) (Figure 4.11) report summarises the distribution of study scores or ATAR Subject Scores for selected studies for a particular year. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school.

For an explanation of box-and-whisker plots, please refer to Appendix 1 – Glossary of Terms.

Figure 4.11


Figure 4.12

The table in Figure 4.12 summarises the distribution of Study Scores for each study for a selected year.

By choosing ‘ATAR Subject Scores’ instead of ‘VCAA Study Scores’, a similar chart and table can be produced to show the distribution of ATAR Subject Scores.

Report 8: Single VCE Study (Study Scores and Graded Assessments)

The Single VCE Study (Study Scores and Graded Assessments) report (Figure 4.13) displays the distribution of Study Scores, ATAR Subject Scores or graded assessments for a selected study in a particular year for the various School Groups relative to This School. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A. In this case, individual scores are shown as dots on the graph.

Schools have the option of further breaking down their results by gender, year level or class group and by school defined data fields.

Users can also access Examination papers and Examiner’s reports for the selected study from the report selection screen and report output screen.

For an explanation of box-and-whisker plots, please refer to Appendix 1 – Glossary of Terms.

Figure 4.13


The table in Figure 4.14 summarises the distribution of Study Scores in a particular year for the various School Groups and This School. In addition, there is the option to also view results for gender and school defined data fields.

Figure 4.14

Report 9: Single VCE Study (Study Scores x Time)

The VCE Study (Study Score x Time) report (Figure 4.15) displays the distribution of Study Scores or ATAR Subject Scores for a study over selected years for the various School Groups and This School. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A.

A gender breakdown of each distribution is optional.

Figure 4.15

For an explanation of box-and-whisker plots, refer to Appendix 1 – Glossary of Terms.


The table in Figure 4.16 summarises the distribution of Study Scores in selected years for the various School Groups and This School.

Figure 4.16


Adjusted Score ReportsThe VCEDS provides the following reports of adjusted scores and estimates:

• Selected VCE Studies (Adjusted)

• Single VCE Studies (Adjusted)

• Single VCE Studies (Adjusted) x Time

• Subgroup Comparison (Adjusted)

• Achieved versus Predicted Scores

Report 10: Selected VCE Studies (Adjusted)

The Selected VCE Studies (Adjusted) report (Figure 4.17) displays adjusted estimates and their associated confidence intervals for selected studies for a particular year. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school.

The graph shows the ability and gender adjusted estimate for each study and the 95% confidence interval for this estimate. Where there is insufficient data, the estimate is not reported and is replaced by N/A.

Figure 4.17


The table in Figure 4.18 displays the ability and gender adjusted estimate and associated 95% confidence interval for each study for a particular year for This School. The reliability of the estimate is also reported.

Figure 4.18

Report 11: Single VCE Studies (Adjusted)

The Single VCE Studies (Adjusted) report (Figure 4.19) summarises the distribution of adjusted estimates for a study in a particular year for the various School Groups relative to This School. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data (showing dots in the graph for relevant school group), the distribution is not reported and is replaced by N/A in the report.


The estimate for This School displays the estimate and the 95% confidence interval for this estimate. Where there is insufficient data, the school estimate is not reported and is replaced by N/A.

Figure 4.19


The table in Figure 4.20 summarises the distribution of adjusted estimates in a particular year for the various School Groups.

Figure 4.20

The table in Figure 4.21 provides the estimate and 95% confidence interval for the estimate for a study in a particular year for This School. The reliability of the estimate is also reported.

Figure 4.20

Report 12: Single VCE Studies (Adjusted) x Time

The Single VCE Studies (Adjusted) x Time report (Figure 4.22) displays the adjusted estimates for a study over selected years for the various School Groups relative to This School.

The graph shows the estimate and the 95% confidence interval for this estimate for a study over selected years. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the estimate is not reported and is replaced by N/A.

Figure 4.22


The table in Figure 4.23 displays the adjusted estimates and accompanying confidence intervals for a study in selected years for the various School Groups and This School.

Figure 4.23


Report 13: Subgroup Comparison (Adjusted)

The Subgroup Comparison (Adjusted) report (Figure 4.24) displays the adjusted estimates adjusted for ability for subgroups of students for a study in a particular year. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the estimate is not reported and is replaced by N/A.

Breakdown by gender, year level or class group of each distribution is optional as are School-defined data fields. The report for each subgroup of students displays the estimate and the 95% confidence interval for this estimate.

Figure 4.24

The table in Figure 4.25 displays the adjusted estimates and accompanying confidence intervals for a study for the school and selected subgroups of students.

Figure 4.25


Report 14: Achieved versus Predicted Scores

The Achieved versus Predicted Scores report (Figure 4.26) displays the distribution of Study Scores and predicted scores for a selected study in a particular year. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A.

Schools have the option of further breaking down their results by gender, year level or class group and by School-defined data fields.


Figure 4.26

The table in Figure 4.27 summarises the distribution of scores for a selected study in a particular year.

Figure 4.27


GAT Score ReportsThe VCEDS provides the following reports of GAT scores:

• GAT and Study Score Distributions

• GAT and Study Score Distributions x Time

Report 15: GAT and Study Score Distributions

The GAT and Study Score Distributions report (Figure 4.28) displays the distribution of standardised GAT scores, GAT component scores, Study Scores and ATAR Subject Scores. The report is available for students enrolled in the school as their Home school, and where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A.


Figure 4.28

The table in Figure 4.29 summarises the distribution of scores for a selected study in a particular year.

Figure 4.29


Report 16: GAT and Study Score Distributions x Time

The GAT and Study Score Distributions x Time report (Figure 4.30) displays the distributions of standardised GAT scores, Study Scores and ATAR Subject Scores over selected years. The report is available for students enrolled in the school as their Home school, and, where relevant, for students enrolled in the school as their Assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A.


Figure 4.30

The table in Figure 4.31 summarises the distribution of scores for selected study over time.

Figure 4.31

Since the GAT scores are now standardised to the same means and standard deviations as Study Scores (mean 30, standard deviation 7), this allows users to compare their students’ achievement directly with their GAT performance.


Student-level ReportsThe VCEDS provides the following reports at the student level:

• Student Results by Study

• Student Results across Studies.

Report 17: Student Results by Study

The Student Results by Study report (Figure 4.32) displays a scatter plot of the achieved and predicted Study Scores for all students in the group. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Schools have the option of further highlighting results for a particular group of students by gender, year level or class group. If the cursor is made to hover over a point on the scatter plot, the name of the student will appear. By focusing on points far above or far below the diagonal line, users can identify exceptional performance.

Using this feature, users may be able to identify individuals or groups of students who have exceeded expectations, or who have fallen short of them.

Figure 4.32

Note that Report 17 contains data that identifies individual students. Schools are provided with such data on a confidential basis for use in planning school improvement. Such information must not be included in any material that is likely to be distributed beyond the school.


The table in Figure 4.33 lists the scores for each student in the group.

Figure 4.33

Report 18: Student Results across Studies

The Student Results across Studies report (Figure 4.34) displays the Study Scores and predicted scores for all studies undertaken by a student in a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 4.34

If a particular student has been identified as an exceptional performer (for better or for worse!) by looking at Report 17, this feature allows the user to enquire whether that performance was unique to that study, or whether a pattern existed across other studies undertaken by that student.

As for Report 17, users should note that Report 18 contains data that identifies individual students. Schools are provided with such data on a confidential basis for use in planning school improvement. Such information must not be included in any material that is likely to be distributed beyond the school.

Appendix 1: Glossary of terms VCE Data Service User Manual 35

Appendix 1: Glossary of terms

Adjusted Score Reports

The VCEDS provides a series of reports that allow users to review estimates that have been adjusted for ability and gender.

Figure 1

There are five reports as per the following.

REPORT DESCRIPTION

Report 10 Selected VCE Studies (Adjusted)

This report displays the distribution of adjusted score estimates for selected studies in a particular year.

Report 11 Single VCE Study (Adjusted)

This report displays the distribution of adjusted score estimates for a selected study in a particular year for your school, and allows you to make comparisons with the various School Groups relative to This School.

Report 12 Single VCE Study (Adjusted) x Time

This report displays adjusted score estimates for a selected study over selected years and allows you to make comparisons with the various School Groups relative to This School.

Report 13 Subgroup Comparison (Adjusted)

This report displays the distribution of adjusted score estimates for a selected study in a particular year for all students in This School, for subgroups of Gender, Year Level and Class Group, and for any School Defined Field groups set up by the school.

Report 14 Achieved versus Predicted Scores

This report displays the distribution of achieved and predicted Study Scores for selected studies in a particular year for all students in This School, for subgroups of Gender, Year Level and Class Group and for any School Defined Field groups set up by the school.

All Schools

‘All Schools’ is defined as all schools that offered a particular VCE study in a chosen year.

36 VCE Data Service User Manual Appendix 1: Glossary of terms

Assessment grades

Assessment grades are reported by the VCAA for each VCE study. All VCE studies have three graded assessments for each Unit 3 and 4 sequence. Each study includes at least one examination, most have School-assessed Coursework, and some have School-assessed Tasks. Every scored VCE VET program has two assessment grades: one for coursework, one for an external exam.

School-assessed Tasks are moderated by the VCAA. Assessment grades are reported on an A+ to E, UG (ungraded) scale. An assessment reported as NA means ‘not assessed’.

In graphical and tabular summaries of assessment grades, they are translated to a 10point scale as follows:

A+ = 10

A = 9

B+ = 8

B = 7

C+ = 6

C = 5

D+ = 4

D = 3

E+ = 2

E = 1

UG = 0

Assessing school

An assessing school is the school responsible for providing the assessment of one or more units for a student. A single student can have one or more assessing schools. The assessing school is usually, but not always the home school.

Australian Tertiary Admission Rank (ATAR) Subject Score

The ATAR Subject Score is a scaled Study Score computed by VTAC to calculate an ATAR score for each student seeking entry to a tertiary course. A table showing the scaling translation of VCAA Study Scores to ATAR Subject Scores is published on the VTAC website each year. This used to be known as the Equivalent National Tertiary Entrance Rank (ENTER) Subject Score.

Box-and-whisker plots

The distribution of scores is indicated by means of box-and-whisker plots in which the box encloses the middle 50 per cent of schools, the top of the box represents the 75th percentile, the bottom of the box the 25th percentile, the top of the whisker the 90th percentile, the bottom of the whisker the 10th percentile. The heavy line in the middle of the box indicates the 50th percentile.


Combined Study Reports

The VCEDS provides a series of reports that allow users to review overall results for the school.

Figure 2

There are six reports as follows.

REPORT DESCRIPTION

Report 1 All VCE Studies

This report displays the distribution of Study Scores for all VCE studies in a particular year for the various School Groups and This School.

Report 2 Assessment Grades

This report displays the percentage of assessment grades for all VCE studies in a particular year for the various School Groups and This School. Reports can be sought for all studies, or any particular Study Group.

Report 3 Study Score Frequencies

This report displays the distribution of Study Scores for all VCE studies in a particular year for the various School Groups and This School. Reports can be sought for all studies, or any particular Study Group.

Report 4 Completions

This report lists the number and percentage of students in the school completing each unit and being awarded Study Scores for a particular year. Reports can be sought for all studies, or any particular Study Group.

Report 5 Study Scores of 40 or more

This report displays the percentage of Study Scores of 40 or above for all VCE studies for selected years for the various School Groups and This School.

Report 6 Region Schools Grouping

List of schools that belong to the same sector and region as This School.

Confidence intervals

All adjusted estimates are associated with a degree of uncertainty. This uncertainty can be quantified and a confidence interval can be constructed around each estimate.

In the VCEDS reports, confidence intervals are reported both in tabular form and graphically. For each estimate there is a lower and upper confidence interval, which are referred to as ‘Low’ and ‘High’ respectively.

Graphically, the confidence intervals are displayed by means of ‘error bars’ around the adjusted estimates. They have been set so that two schools are judged to have significantly different ability and gender adjusted estimates (at the 5 per cent significance level) only if their error bars do not overlap.


The following example shows the adjusted estimates of two schools, A and B, and their accompanying error bars.

Figure 3

While the estimate of School B is lower than that of School A, the error bars around the two estimates overlap, and so it is not possible to say (at the 5 per cent significance level) that the estimates are significantly different.

It will be observed that, all things being equal, adjusted estimates for studies in which the school has small numbers of enrolled students are associated with large error bars. The opposite is also true: adjusted estimates for studies in which the school has large numbers of enrolled students (for example English) typically have small error bars.

All confidence intervals are based on analyses carried out within a given study. This needs to be borne in mind when comparing adjusted estimates for different studies. In such cases, the confidence intervals should be used as a guide only, since they have not been derived from a multistudy analysis.

Confidence intervals provide the most convenient way of expressing the degree of uncertainty associated with ability- and gender-adjusted estimates. Another method is to express this degree of uncertainty as a coefficient of reliability. Information on the reliability of adjusted estimates for each school is provided in the relevant VCEDS reports.

General Achievement Test (GAT)

The GAT is an essential part of the VCE assessment procedure. Although GAT results do not count directly towards VCE results, they do play an important role in checking that School-based Assessments and examinations have been accurately assessed. The VCAA will use GAT scores in:

• the statistical moderation of School-based Assessments

• checking the accuracy of student scores in examinations

• the calculation of a derived exam score (DES).

The GAT provides measures of the general knowledge and skills that students have built up during their school years in three areas:

• Written Communication

• Mathematics, Science and Technology, and

• Humanities, the Arts and Social Science.

These skills are built up over a period of time prior to Year 12 and provide an indication of how well students are equipped to undertake VCE studies. For this reason, the GAT is a good predictor of VCE performance and can be used as an indicator of where students would be in their VCE studies ‘other things being equal’. The GAT is therefore regarded as a measure of general ability or preparedness for the VCE.

Figure 3


The GAT has been shown to be a good predictor of performance in most VCE studies. The exceptions are those studies that depend mostly on abilities other than those measured by the GAT, including subjects with a substantial practical or performance component.

GAT Score Reports

The VCEDS provides a series of reports that allow users to examine the distribution of GAT scores for one or more years, and comparing the Study Scores to the GAT scores.

Figure 4

There are two reports as follows.

REPORT DESCRIPTION

Report 15 GAT and Study Score Distributions

This report displays the distribution of GAT scores, component GAT scores, Study Scores and ATAR Subject Scores for a particular year.

Report 16 GAT and Study Score Distributions x Time

This report displays the distribution of GAT scores, Study Scores and ATAR Subject Scores for selected years.

High

In the VCEDS, confidence intervals are reported for all adjusted estimates. For each estimate there is a lower and upper confidence interval, which are referred to as ‘Low’ and ‘High’ respectively.

Home School

A student’s home school is the school at which that student is enrolled. It is the school responsible for all student data and course enrolments. A student may only have one home school at a time.

Sometimes students attend another school for one or more of their studies, for example when there are insufficient enrolments in their home school for the study to be made available. The other school is the student’s assessing school for those studies.

Low

In the VCEDS, confidence intervals are reported for all adjusted estimates. For each estimate there is a lower and upper confidence interval, which are referred to as ‘Low’ and ‘High’ respectively.


Multilevel modelling

Analysing student performance has always presented researchers and statisticians with a difficult challenge. Students are not randomly assigned to classes or schools and, as a consequence, performance is very much affected by the composition of the students in a given school or classroom. Over the past decade or more, researchers have developed new methods for analysing data in which subjects are nested within groups at two or more levels (for example within classes within schools). These methods employ what is known as multilevel or hierarchical linear modelling.1

Multilevel modelling provides the most statistically valid way of obtaining adjusted scores and accompanying confidence intervals. To understand multilevel models, it is useful to begin by considering a simple (single-level) regression model. The formula for a simple regression model is shown in the following equation (1):

yi = b0 + b1xi + ei (1)

The subscript i takes values from 1 to n, and n is the total number of students in the sample of schools. For the i-th student, yi is the outcome variable (say their VCE Study Score) and xi is the control variable (say performance on one of the subtests of the GAT).

A student’s predicted score (ŷ) is given by the following equation (2):

ŷi = b0 + b1xi (2)

Here ŷi is the predicted VCE Study Score for the i-th student, b0 is the intercept where the regression line meets the vertical axis, and b1 is the slope.

In equation (1), ei is the difference between the i-th student’s actual VCE Study Score and the predicted score. It is referred to as a ‘residual’ because it is that part of the score that is not predicted by the fixed part of the model represented by equation (2).

Moving now to a simple multilevel model, we can rewrite equation (1) as the following equation (3):

yij = b0 + b1xij + (uj + eij) (3)

In this equation terms are denoted by two subscripts, one indicating the student and the other the school attended by the student. The critical feature of this equation (3) is the uj term, which is the difference between the j-th school’s actual intercept and the overall mean value. It is what is known as a level-2 residual.

It is the existence of the two residuals – the level-2 school residual and the level-1 student residual – that identifies equation (3) as a multilevel model. Furthermore, it is through the estimation of level-2 residuals that it is possible to obtain adjusted estimates.

Let us now consider the model used as part of the VCEDS to provide estimates adjusted for ability and gender. The simple model of equation (3) can now be expanded in greater detail as follows:

yij = b0 + b1xlij + b2x2j + b3x4j + b5x5ij + (u0j+e0ij)

Student Score Predicted Score School residual Student residual

(4)

1 Several specialised computer software packages are now available to undertake multilevel regression analysis. The best known include HLM (Bryk, Raudenbush & Congdon) and MLN/MLwiN (Rasbash & Woodhouse, 1995; Goldstein, et al., 1998). In addition, modules to undertake restricted kinds of multilevel modelling have been incorporated into a number of general purpose software packages, including SAS and SPSS. For a review of currently available software, the reader is referred to Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London/ Thousand Oaks/ New Delhi: Sage Publications.


In Equation (4) the terms have the following meanings:

yij = the score on the outcome variable (VCE Study Score) for student i in school j.

b0 = an overall constant term, indicating the average value for male students, adjusting for the effects of the three subcomponents of the GAT and the school average GAT.

b1x1ij = the slope coefficient and score on the Written Communication subtest (GATCOM) for student i in school j, indicating the effect of GATCOM.

b2x2j = the slope coefficient and score on Mathematics, Science and Technology subtest of the GAT (GATMST) for student i in school j, indicating the effect of GATMST.

b3x3ij = the slope coefficient and score on the Arts, Social Science and Humanities subtest of the GAT (GATHU) for student i in school j, indicating the effect of GATHU.

b4x4j = the slope coefficient and average score of the cohort of students within the school on a composite GAT score for school j, indicating the effect of the average academic ability of students in school j.

b5x5ij = the slope coefficient and dummy variable for gender (scored ‘0’ for male students and ‘1’ for female students) for student i in school j, indicating the effect of being a girl.

u0j = a residual term for the intercept of school j, indicating the unexplained effect due to the school attended by student i.

e0ij = a residual term for student i in school j, indicating that part of the student’s performance that could not be explained by the control variables.

It is the school residual term (u0j) that is the estimate adjusted for ability and gender for each school.

These scores are centred on 0. If a school performs precisely as predicted, given students’ abilities and their gender composition, the school will have an estimate of zero. If the students perform less well than expected, the school will have a negative estimate, and if they perform better than expected, the school will have a positive estimate.

N/A

N/A reported on a VCEDS report indicates that there was insufficient data for that component of the report.

For reports relating to Study Scores and grades, N/A is reported where there are less than 10 schools with Study Scores or grades in the reported category.

For reports relating to school-adjusted estimates, estimates have not been generated for schools for a given study where there is:

• less than 400 total enrolments with complete data

• less than 20 schools with enrolments

• insufficient correlation between study and GAT scores.

In these cases, the reliability of such adjusted estimates would be unacceptably low.

Where school-adjusted estimates have been generated:

• no school estimate will be produced for the school where the number of students is less than five and N/A will be reported.

• No distribution of school ability and gender-adjusted estimates will be displayed where there are less than 10 school-adjusted estimates and N/A will be reported.


Percentile

A percentile is a measure used in statistics indicating the value (e.g. Study Score) below which a given percentage of observations in a group of observations fall. For example, the 25th percentile is the value below which 25 per cent of the observations may be found.

School-defined data fields

The VCEDS provides schools with the option of defining up to 10 fields of information about students (for example Campus, English as a Second Language, Language other than English spoken at home). Each information field can have up to nine categories.

School-defined data fields and data categories are created, edited and deleted using the VCEDS School Administration Site.

For example

Campus Campus 1 Campus 2

ESL ESL Non-ESL

Language spoken at home English Greek Italian

Turkish

Once school-defined data fields have been created, they can be used to make further within-school comparisons.

School Groups

There are many options to choose from in the reports. One of the options is to compare the results of This School with different School Groups. By selecting one or more School Groups, the results for This School can be compared with All Schools, This School Sector, and/or This School Region.

This School refers to the user’s school.

All Schools refers to all schools in Victoria.

This School Sector refers to all schools in the same sector as that of the user’s school.

This School Region refers to all schools in the same sector and region as that of the user’s school. Only Catholic and government sectors have education regions for their school. Hence, This School Region is only available as an option in the reports for Catholic and government schools. For a list of schools in your region, please refer to Report 6. Note also that This School Region option is available in Reports 1, 2, 3, 5, 8, 9, 11 and 12.


Study Group (SG)

Each study has been allocated to a Study Group according to their discipline. For example, the English study group contains the studies English, English as an Additional Language (EAL), English Language and Literature.

There are 10 Study Groups:

• Business Studies

• English

• Health and Physical Education

• Humanities

• Mathematics

• Science

• Technology

• The Arts

• VCE Languages

• VET in Schools.

Single Study Reports

The VCEDS provides a series of reports that allow users to review results for individual studies.

Figure 5

There are three reports as follows.

REPORT DESCRIPTION

Report 7 Selected VCE Studies (Study Scores)

This report displays the distribution of Study Scores or ATAR Subject Scores for a study or SG in a particular year.

Report 8 Single VCE Study (Score Scores and Graded Assessments)

This report displays the distribution of Study Scores, ATAR Subject Scores or assessment grades for a study in a particular year for All Schools, schools in this School Sector and This School Region, relative to This School, for subgroups of Gender, Year Level and Class Group and for any School Defined Field groups set up by the school.

Report 9 Single VCE Study (Study Scores x Time)

This report displays the distribution of Study Scores or ATAR Subject Scores for a study over selected years for All Schools, schools in this School Sector and This School Region relative to This School.


Study Scores

Study Scores are calculated by the VCAA by combining external and internal assessment results into one score for the study. Study Scores are reported on a scale of 0–50 and have a mean of 30 and a standard deviation of 7 across the state.

Student-level Reports

The VCEDS provides a series of reports that allow users to review the performance of individual students and patterns of performance against expectation for individual studies.

Figure 6

There are two reports as follows.

REPORT DESCRIPTION

Report 17 Student Results by Study

This report displays a scatter plot of student actual and predicted scores for a study in a particular year. The user can highlight subgroups of students on the scatter plot.

Report 18 Student Results across Studies

This report displays the actual and predicted scores for all studies undertaken by a selected student in a particular year.

This School

This School refers to the user’s school.

This School Region

It refers to all schools in the same sector and region as that of the user’s school. This School Region is only available as an option in the reports for Catholic and government schools.

For government schools, the regions are the four DEECD regions: North-Eastern Victoria, North-Western Victoria, South-Western Victoria and South-Eastern Victoria.

For Catholic schools the regions are the three regional Archdioceses (Sale, Sandhurst and Ballarat) and four Metropolitan Regions (Eastern, Northern, Western and Southern).

For a list of schools in your region, refer to Report 6. Note that This School Region option is available in Reports 1, 2, 3, 5, 8, 9, 11 and 12.

This School Sector

Schools in This School Sector are defined as all schools from the same school sector as that of the user’s school, offering a particular study in a chosen year.


VCE

The Victorian Certificate of Education (VCE) is awarded on behalf of the State Government of Victoria to students upon successful completion of Years 11 and 12 and allows entry to tertiary institutions.

VCE Study

The VCEDS provides reports for all VCE studies where there is sufficient data not to report N/A. Following is a complete list of scored VCE studies and their classification in terms of Study Group. Please note that some studies have ceased, but are still made available in the VCE Data Service for time-series comparisons.

STUDY CODE VCE STUDY STUDY GROUP

AC03 Accounting Business Studies

AH03 Agricultural and Horticultural Studies Technology

LO41 Albanian Languages Other Than English

LO02 Arabic VCE Languages

LO44 Armenian VCE Languages

AR03 Art The Arts

IM05 Arts (Multimedia) Arts VET in Schools

IM03 Arts (Multimedia) Creative VET in Schools

IM04 Arts (Multimedia) Enterprise VET in Schools

LO03 Auslan VCE Languages

PS03 Australian Politics Humanities

AS02 Australian Studies Humanities

LO35 Bengali Languages Other Than English

BI03 Biology Science

LO50 Bosnian VCE Languages

BU18 Business VET in Schools

BU03 Business (Office Admin) VET in Schools

BU02 Business (Office Admin) NAP Scored Strm VET in Schools

BU05 Business Administration VET in Schools

BU10 Business Administration NAP Scored Strm VET in Schools

BM03 Business Management Business Studies

CH03 Chemistry Science

LO04 Chinese First Language VCE Languages

LO39 Chinese Second Language VCE Languages

LO48 Chinese Second Language Advanced VCE Languages

LO01 Classical Greek VCE Languages

LO51 Classical Hebrew VCE Languages

CS03 Classical Studies Humanities

CT17 Community Services VET in Schools

CT07 Community Services (Community Work) VET in Schools

CT06 Community Services (CW) NAP Scored Strm VET in Schools

CT11 Community Services Work VET in Schools

CA03 Contemporary Australian Society Humanities

CN03 Contemporary Society Humanities

LO05 Croatian VCE Languages



LO06 Czech Languages Other Than English

DA03 Dance The Arts

DN04 Dance (VCE VET) VET in Schools

DS03 Dance Styles The Arts

DR03 Drama The Arts

LO07 Dutch VCE Languages

EC03 Economics Business Studies

ET03 Electronics VET in Schools

ET06 Electrotechnology VET in Schools

EG07 Engineering Studies (Certificate II) VET in Schools

EG08 Engineering Studies (Certificate III) VET in Schools

EN01 English English

EN09 English (EAL) English

EL01 English Language English

EV03 Environmental Science Science

ES03 Environmental Studies Humanities

EQ04 Equine Industry VET in Schools

LO08 Estonian Languages Other Than English

LO45 Filipino VCE Languages

FS02 Financial Services (VCE VET) VET in Schools

FY03 Food and Technology Technology

LO09 French VCE Languages

FN16 Furnishing VET in Schools

FN03 Furnishing (Cabinet Making) VET in Schools

MA07 Further Mathematics Mathematics

GE03 Geography Humanities

LO10 German VCE Languages

PS05 Global Politics Humanities

GC03 Graphic Communication The Arts

LO22 Greek VCE Languages

HH03 Health and Human Development Health and Physical Education

HE03 Health Education Health and Physical Education

LO11 Hebrew VCE Languages

LO36 Hindi VCE Languages

HI15 History: Asian History Humanities

HI08 History: Australian History Humanities

HI10 History: History of Western Ideas Humanities

HI11 History: Koorie History Humanities

HI16 History: Renaissance Italy Humanities

HI13 History: Revolutions Humanities

HI14 History: The City in History Humanities

HS13 Hospitality VET in Schools

HS07 Hospitality (Commercial Cookery) VET in Schools

HS06 Hospitality (Food & Beverage) VET in Schools



HS14 Hospitality (Kitchen Operations) VET in Schools

HS03 Hospitality (Ops) NAP CC Scored Strm VET in Schools

HS02 Hospitality (Ops) NAP F&B Scored Strm VET in Schools

HD03 Human Development Health and Physical Education

LO12 Hungarian VCE Languages

LO38 Indigenous Languages of Victoria VCE Languages

LO13 Indonesian First Language VCE Languages

LO40 Indonesian Second Language VCE Languages

IE03 Industry and Enterprise Business Studies

IN13 Information Technology (General) VET in Schools

IN14 Information Technology (Network Admin) VET in Schools

IN12 Information Technology (Software Apps) VET in Schools

IN21 Information Technology (VCE VET) VET in Schools

ET11 Integrated Technologies VET in Schools

MU05 Interactive Digital Media VET in Schools

IS03 International Studies Humanities

IT02 IT Applications Technology

LO14 Italian VCE Languages

LO15 Japanese First Language VCE Languages

LO46 Japanese Second Language VCE Languages

LO16 Khmer VCE Languages

LO37 Korean First Language VCE Languages

LO47 Korean Second Language VCE Languages

LB20 Laboratory Skills VET in Schools

LB18 Laboratory Skills (VCE VET) VET in Schools

LO17 Latin VCE Languages

LO18 Latvian Languages Other Than English

LS03 Legal Studies Business Studies

LI01 Literature English

LO19 Lithuanian Languages Other Than English

LO20 Macedonian VCE Languages

LO21 Maltese VCE Languages

MT03 Materials and Technology Technology

MA08 Mathematical Methods Mathematics

MA11 Mathematical Methods (CAS) Mathematics

ME03 Media The Arts

MU02 Multimedia VET in Schools

MI13 Music (VCE VET) VET in Schools

MC03 Music Group Performance The Arts

MI09 Music Industry (Technical Production) VET in Schools

MC05 Music Investigation The Arts

MC04 Music Performance The Arts

MC02 Music Solo Performance The Arts

MS03 Music Style and Composition The Arts



MI15 Music: Technical Production (VCE VET) VET in Schools

NP03 National Politics Humanities

OS03 Outdoor and Environmental Studies Health and Physical Education

OE03 Outdoor Education Health and Physical Education

LO32 Persian VCE Languages

PL03 Philosophy Humanities

PE03 Physical Education Health and Physical Education

PH03 Physics Science

PH09 Physics Pilot Science

LO23 Polish VCE Languages

LO33 Portuguese VCE Languages

DT03 Product Design and Technology Technology

PY03 Psychology Science

LO49 Punjabi VCE Languages

RE03 Religion and Society Humanities

LO42 Romanian VCE Languages

LO24 Russian VCE Languages

SC03 Science Science

LO25 Serbian VCE Languages

LO34 Sinhala VCE Languages

LO26 Slovenian Languages Other Than English

SO03 Sociology Humanities

IT03 Software Development Technology

LO27 Spanish VCE Languages

MA09 Specialist Mathematics Mathematics

SR27 Sport & Recreation VET in Schools

SR19 Sport & Recreation (Outdoor Recreation) VET in Schools

SR21 Sport & Recreation (Community Recreation) VET in Schools

SA03 Studio Arts The Arts

LO28 Swedish VCE Languages

ST03 Systems and Technology Technology

SE03 Systems Engineering Technology

LO43 Tamil VCE Languages

TD03 Technical Design & Development Technology

TT03 Texts and Traditions Humanities

TS03 Theatre Studies The Arts

LO29 Turkish VCE Languages

LO30 Ukrainian VCE Languages

LO31 Vietnamese VCE Languages

VC03 Visual Communication Design The Arts

LO52 Yiddish VCE Languages

Appendix 2: Generating VCEDS reports VCE Data Service User Manual 49

Appendix 2: Generating VCEDS reports

Report 1: All VCE Studies

The All VCE Studies report (Figure 1) displays the distribution of Study Scores or ATAR Subject Scores for all VCE studies in a particular year for This School and allows for comparison with the various School Groups. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school. A gender and year level breakdown of each distribution is optional.

Figure 1

• Select Year by choosing it from the drop-down box.

• Select the desired comparison group/s of schools (All Schools, schools in This School Sector and This School Region).

• Click next to Gender or Year Level if a subgroup analysis is required.

• The default selection for data is Home School Data. If additional data exists for any study for the selected year for This School as an assessing school, a second option Assessing School Data will appear and can be selected.

• Click VCAA Study Score or ATAR Subject Score as the required assessments.

• Click Report to generate the report.

• Click Reset if you wish to clear report selections.

50 VCE Data Service User Manual Appendix 2: Generating VCEDS reports

Report 2: Assessment Grades

The Assessment Grades report (Figure 2) displays the percentage of assessment grades for all VCE studies in a particular year for This School and allows for comparison with with the various School Groups. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 2


• Select the desired comparison group/s of schools (All Schools, schools in This School Sector, This School Region).


• Click next to All VCE Studies or next to the desired Study Group to select a group of studies.




Report 3: Study Score Frequencies

The Study Score Frequencies report (Figure 3) displays the distribution of Study Scores for all VCE studies in a particular year for This School and allows comparison with with the various School Groups. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 3








Report 4: Completions

The Completions report (Figure 4) lists the number and percentage of students in the school who were completing each unit and awarded Study Scores in a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 4


• Click next to All VCE Studies or next to the desired Study Group/s to select a group of studies or a number of groups of studies.





Report 5: Study Scores of 40 and above

The Study Scores of 40 and above report (Figure 5) displays the percentage of Study Scores of 40 or above for all VCE studies for selected years for the various School Groups. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 5

• Select Year/s by clicking next to the required year/s.







Report 6: Region Schools Grouping

The Region Schools Grouping report (Figure 6) provides a list of schools that belong to the same sector and region as This School. This report only applies to Catholic and government schools. For a more detailed explanation please refer Appendix 1 – Glossary of Terms.

Figure 6


Report 7: Selected VCE Studies (Study Scores)

The Selected VCE Studies (Study Scores) report (Figure 7) summarises the distribution of Study Scores or ATAR Subject Scores for selected studies for a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 7

• Select Year by choosing it from the drop-down box. When the year is selected, the list of Studies Offered will be updated for the selected year.

• Select studies by highlighting the study name that is to be selected. Hold the Shift button down to select a group of studies or the Ctrl button down to select multiple studies. Alternatively, select groups of studies by clicking next to All VCE Studies or next to the desired Study Group/s.

• The default selection for data is Home School Data. If additional data exists for any study for the selected year for This School as an assessing school, a second option, Assessing School Data, will appear and can be selected.

• Select the desired assessments from VCAA Study Score or ATAR Subject Score.




Report 8: Single VCE Study (Study Scores and Graded Assessments)

The Single VCE Study (Study Scores and Graded Assessments) report (Figure 8) displays the distribution of Study Scores, ATAR Subject Scores or assessments for a selected study in a particular year for the various School Groups relative to This School. The report is available for students enrolled in the school as their home school, and where relevant for students enrolled in the school as their assessing school.

Figure 8



• Select the desired comparison group/s of school-defined data fields if such data has been recorded in the VCEDS.


• Click next to Gender, Year Level and/or Class if a subgroup analysis is required.

• Highlight the study name that is to be selected. Hold the Shift button down to select a group of studies or the Ctrl button down to select multiple studies.

• Select the desired assessments from VCAA Study Score, ATAR Subject Score or Graded Assessments. If required, click on the hyperlink to review relevant Examination papers and Examiner’s reports.




Report 9: Single VCE Study (Study Scores x Time)

The Single VCE Study (Study Scores x Time) report (Figure 9) displays the distribution of Study Scores or ATAR Subject Scores for a study over selected years for the various School Groups. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 9

• Select Year/s by clicking next to the required year/s. When each year/s is selected, the list of Studies Offered will be updated for the selected years.



• Select the desired assessments from VCAA Study Score and ATAR Subject Score.





Report 10: Selected VCE Studies (Adjusted)

The Selected VCE Studies (Adjusted) report (Figure 10) displays adjusted estimates and associated confidence intervals for selected studies for a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 10


• Select studies by highlighting the study name that is to be selected. Hold the Shift button down to select a group of studies or the Ctrl button down to select multiple studies. Alternatively, select groups of studies by clicking next to All VCE Studies or next to the desired Study Group/s.





Report 11: Single VCE Study (Adjusted)

The Single VCE Study (Adjusted) report (Figure 11) summarises the distribution of adjusted estimates for a study in a particular year for the various School Groups relative to This School. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 11


• Select the desired comparison group/s of schools (All Schools, schools in This School Sector).






Report 12: Single VCE Study (Adjusted) x Time

The Single VCE Study (Adjusted) x Time report (Figure 12) displays the adjusted estimates for a study over selected years for the various School Groups.

Figure 12

• Select Year/s by clicking next to the required year/s. When each year is selected, the list of Studies Offered will be updated for the selected years.







Report 13: Subgroup Comparison (Adjusted)

The Subgroup Comparison (Adjusted) report (Figure 13) displays the adjusted estimates adjusted for ability for subgroups of students for a study in a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 13


• Select the desired comparison group/s of school defined data fields if such data has been recorded in the VCE Data Service.

• Click next to Gender; Year Level and/or Class Group if a subgroup analysis is required.






Report 14: Achieved versus Predicted Scores

The Achieved versus Predicted Scores report (Figure 14) displays the distribution of Study Scores and predicted scores for subgroups of students for a selected study in a particular year. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 14


• Highlight the study name that is to be selected. Hold the Shift button down to select a group of studies or the Ctrl button down to select multiple studies. Alternatively, select groups of studies by clicking next to All VCE Studies or next to the desired Study Group/s.

• Click next to Gender; Year Level and/or Class Group if a subgroup analysis is required.





Report 15: GAT and Study Score Distributions

The GAT and Study Score Distributions report (Figure 15) displays the distribution of GAT scores, GAT component scores, Study Scores and ATAR Subject Scores. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school.

Figure 15







Report 16: GAT and Study Score Distributions x Time

The GAT and Study Score Distributions x Time report (Figure 16) displays the GAT scores, Study Scores and ATAR Subject Scores over selected years. The report is available for students enrolled in the school as their home school, and where relevant, for students enrolled in the school as their assessing school. Where there is insufficient data, the distribution is not reported and is replaced by N/A.

Figure 16

• Select Year/s by clicking next to the required year/s. When each year is selected, the list of Studies Offered will be updated for the selected years.






Report 17: Student Results by Study

The Student Results by Study report (Figure 17) displays a scatter plot of the Study Scores and predicted scores for each student in the group.

Figure 17



• To highlight a subgroup of students click one of the student types (Female, Male, Year 11, Year 12 or Class Group). If one clicks Class Group, the screen will refresh with a list of relevant classes for the selected study. Select one or more classes to be highlighted in the report.




Note: Report 17 contains data that identifies individual students. Schools are provided with such data on a confidential basis for use in planning school improvement. Such information must not be included in any material that is likely to be distributed beyond the school.


Report 18: Student Results across Studies

The Student Results across Studies report (Figure 18) displays the Study Scores and predicted scores for all studies undertaken by a student in a particular year.

Figure 18



• Highlight the student name that is to be selected. Hold the Shift button down to select a group of students or the Ctrl button down to select multiple students.



Note: Report 18 contains data that identifies individual students. Schools are provided with such data on a confidential basis for use in planning school improvement. Such information must not be included in any material that is likely to be distributed beyond the school.

Appendix 3: Report short-cut guide VCE Data Service User Manual 67

Appendix 3: Report short-cut guide

68 VCE Data Service User Manual Appendix 3: Report short-cut guide

Rep

ort

fea

ture

s

1. M

easu

re2.

Tim

e3.

Stu

die

s4.

C

om

par

e w

ith

Oth

er

scho

ols

(A

ll,

Sec

tor)

5. C

oho

rt6.

Bas

is o

f A

sses

smen

t7.

Dat

a D

isp

lay

typ

e

VC

E D

S R

epo

rt

VCAA

St

udy

Scor

e

ATAR

Su

bj

Scor

eGA

Gr

ades

GAT

Adj.

Stud

y Sc

ores

One

Year

Mul

t. Ye

arSi

ngle

St

udy

SG

Com

pare

M

ult.

Stud

ies

Sex

Yr

Lvl.

Clas

s Gr

oup

Scho

ol

Defin

ed

Fiel

dsHo

me

Sch.

Asse

ss

Sch.

Tab.

B &

W

PHi

st.

A E

&

E B

S.

Plot

Stu

dy

Sco

res

1Y

YY

YY

YY

YY

Y

3Y

YY

YY

YY

7Y

YY

YY

YY

YY

Y

8Y

YY

YY

YY

YY

YY

YY

Y

9Y

YY

YY

YY

YY

%40

+ s

tud

y sc

ore

s5

YY

YY

YY

YY

Uni

t 3–

4 co

mp

leti

ons

4Y

YY

YY

Y

GA

- G

rad

es2

YY

YY

YY

YY

Ad

just

ed S

tud

y S

core

s

10Y

YY

YY

YY

YY

11Y

YY

YY

YY

Y

12Y

YY

YY

YY

Y

13Y

YY

YY

YY

YY

Y

14A

PY

YY

PY

YY

YY

YY

Y

GA

T R

epo

rt15

YY

YY

YY

YY

YY

16Y

YY

YY

YY

YY

Y

Ind

ivid

ual S

tud

ents

Rep

ort

17Y

YY

YY

YY

YY

YY

18Y

Y*

YY

Y

Sch

oo

l Reg

ion

Gro

ups

6Y

Y

Not

es: Y

* –

only

last

tw

o ye

ars

to c

hose

from

, P –

Par

tially

, GA

– G

rad

ed A

sses

smen

ts, A

P –

Dis

pla

ys A

ctua

l and

Pre

dic

ted

Sco

res,

Tab

– T

able

s, B

& W

P –

Box

and

Whi

sker

Plo

t, H

ist

– H

isto

gram

, A E

& E

B –

Ad

just

ed E

stim

ates

and

Err

or b

and

s, S

. Plo

t –

Sca

tter

Plo

t

Report Short-cut Guide

Appendix 3: Report short-cut guide VCE Data Service User Manual 69

Report Short Cut Guide – Which report do I use?

No. Reason for Report Use

Combined Study Reports – How were the school’s overall results this year?

1 What was the overall level of performance? How does it compare with schools in general or with schools in your sector? How does it compare with results achieved in your school in previous years?

2 What is the overall distribution of grades last year? Did you have the number of high grades, or low grades that you expected? Were there more ‘NA’s and ‘UG’s than you would expect?

3 You might, out of curiosity, like to examine just how many scores of 50, 49, 48, etc. were achieved in a given year. This will be useful if your school focuses on targets around very high Study Scores.

4 Are students completing the basic requirements satisfactorily? Are students getting through, even those who are not high flyers? Are these results improving compared to previous years?

5 Are you producing the number of very high achievers that you would like to? Are there more or less of them than in previous years?

School Region Report – Where does your school fit into its Region Schools Group?

6 Some of the reports provide an option that allows the results of This School to be compared with schools in This School Region. The Region Schools Grouping report provides a list of schools that belong to the same sector and region as This School. This report only applies to Catholic and government schools.

Single Study Reports – What was the overall level of results for particular studies?

7 This allows you to look at results across studies, and to identify studies that you may want to look at more closely.

8 This report provides you with a detailed look at results in a single study, including performance on each graded assessment. You may look at the performance of different groups of students in the study, including gender groups, class groupings, Year 11 and Year 12 students, and other groupings.

9 This report allows you to look back at the pattern of results in any study over many years. Are this year’s results part an encouraging trend, part of a worrying trend, or a ‘blip’ on the radar screen?

Adjusted Study Score Reports – How did our results compare to reasonable expectations?

10 This allows you to look at adjusted estimates across studies, and to identify results that have been significantly higher or lower than you expected given the student cohorts in those studies.

11 This report provides you with a detailed look at the adjusted estimates in a single study, and compares these to the performance of students in other schools. This helps you to view your own students’ achievements in the wider context of other students in other schools across the state.

12 This report allows you to look back at the pattern of adjusted estimates results in any study over several years. If there is a change in the pattern of results, is it explainable in terms of changes in the student cohort, or is there something else happening?

13 This report provides you with a detailed look at the adjusted estimates in a single study. You may look at the performance of different groups of students in the study, adjusted for gender and ability, including class groupings, Year 11 and Year 12 students, and others.

14 What was the pattern of results that you might have expected in each study, and what was the pattern of results that your students actually achieved?

GAT Score Reports – Is our student cohort changing over time?

15 What was the pattern of ability of students in your school, or in any given study, as indicated by the three GAT component scores and the GAT total score? How do the patterns of Study Scores and ATAR Subject Score compare with these?

16 Is there a trend in the overall pattern of GAT results in your school over time? Are there trends in individual studies? And, importantly, if there are such patterns, do the patterns of results that your students have achieved mirror or defy these trends?

Student Level Reports – What can we learn from patterns in the performance of individual students?

17 Look at the pattern of achieved and expected scores in each study individually. Where do particular groups of students appear in that pattern? Who were the exceptions, and how well do you understand the reasons for their exceptionality?

18 Where an individual student has excelled or fell short in comparison with expectations, was it across the board, or was it just in a single study? What can we learn from the results of these students?

vce data service users manual version 5 - weebly...gat will be influenced by what the school has...

Documents