becta ict research conference – june 2002 intro survey details secondary surveys conducted july...

BECTa ICT Research Conference – June 2002

IntroIntro

Survey Details Secondary Surveys conducted July 2000 and June/July 2001 Sponsored by Fischer Family Trust and RM 3500 replies from secondary subject departments in over 2500

schools Total of over 25,000 ratings of individual ICT resources Pilot Primary Survey (180 schools) in 2001

Publications High Impact ICT Resources

Fischer Family Trust Reports 2002 www.fischertrust.org

Further Developments ‘Value-Added’ Analysis Workshop – ‘Making a difference with ICT’ Survey 2002


PhilosophyPhilosophy Ask the experts

Teachers who have been using software in real environments

Key Questions Is it easy to use, motivating etc. useful, but what really matters is “Does it help to improve learning”

IMPACT is more important than LEVEL OF USE

Overall Aim Is to provide feedback to teachers managers in schools about

which ICT resources are felt to have the greatest impact upon pupils’ learning


SurveysSurveys

Survey asked Teachers to provide: a list of the main ICT resources used in the teaching of their

subject(s) - software packages, website, specialist peripherals an overall rating for the impact of ICT on pupils’ learning in

each subject ratings, in terms of use and impact on pupils’ learning, for

each ICT resource An overall rating for the impact of website use (2001) a brief summary of packages used for administration (2001) any other comments

Ratings used were: 1 = Very Little 2 = Some 3 = Significant 4 = Substantial


Overall SummaryOverall Summary

Schools were a representative sample: Distribution (geographical, type, catchment area) similar to national

patterns Comments indicated that levels of resourcing varied significantly,

indicating that responses were not primarily from well-resourced schools.

Less than 10% of secondary departments made very little use of ICT

Impact rated as SIGNIFICANT or SUBSTANTIAL by:

Key Stage 2000 Survey 2001 Survey

1 59%

2 67%

3 40% 49%

4 53% 59%


Variations between SubjectsVariations between Subjects

Impact rated as SIGNIFICANT or SUBSTANTIAL by:

Subject Secondary Primary

Art 55% 38%

D&T 78% 30%

English 46% 79%

Geography 36% 39%

History 23% 51%

ICT 85% 87%

Mathematics 31% 75%

MFL 30%

Music 76% 19%

PE 25% 7%

RE (48%) 24%

Science 38% 54%

Welsh (45%) (29%)

Secondary based upon:

Art 260D&T 360English 168Geography 186History 147ICT 370Mathematics348MFL 272Music 420PE 301RE 28Science 232Welsh 47

Primary based upon 180 schools


Value Added Analysis - Background

Value Added Analysis - Background

Purpose To ask the question “Is there any evidence for the impact of

ICT upon pupils’ progress and attainment”

Methodology Link pupil-level value-added data (KS2-3 and KS3-4 for 2 and 3

years respectively) to software survey responses. Note – performance data provided by government agencies

for the purpose of this research is confidential. Data for individual schools will not be disclosed or published.

Analyse data to see whether pupils make better progress in schools where ICT rated as HIGH IMPACT (Significant or Substantial).


Analyses – School Level CorrelationAnalyses – School Level Correlation

R 2 = 0.90

3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0

Estimated KS3 Mean Level (from KS2 data)

Ac

tua

l K

S3

Me

an

Le

ve

l

R 2 = 0.90

2.00

3.00

4.00

5.00

6.00

7.00

8.00

2.00 3.00 4.00 5.00 6.00 7.00 8.00

Estimated GCSE Mean Score (from KS3 Data)

Ac

tua

l G

CS

E M

ea

n S

co

re

KS2->KS3

KS3->KS4

Graphs plot Actual vs Estimated Mean Score

Each graph shows data for ~10,000 schools over a 3 year period


Analyses - AccuracyAnalyses - Accuracy

Correlations Inputs:

TA and Test Levels in each subject (Pupil Level) Gender (Pupil Level) Free School Meals Entitlement (School Level)

Outputs: Total Points Score

Key Stage Pupils Schools PUP SCH LEA

1->2 0.1 million 4,000 / 3 yrs 0.82 0.75 NA

2->3 1.7 million 12,000 / 3 yrs 0.87 0.95 0.93

3->4 2.2 million 12,000 / 3 yrs 0.89 0.95 0.87


VA for ‘HIGH Impact’ Secondary Departments

Consistently

Better

Better

in 2000

Improving

Faster

Little

Difference

Art Design & Technology English Geography History Mathematics MFL Music Physical Education Science

Value Added AnalysisValue Added Analysis


FAQFAQ

Are the outcomes simply a reflection of overall school differences?

Using pupil-level value-added data means that we can control for pupils’ prior attainment

Detailed analysis shows that ‘high impact’ departments tend to achieve better results than other departments in the same school

Does this PROVE that ICT use has a positive impact upon standards?

NO – but it does show that pupils make better progress in departments where ICT is felt to be making a significant contribution to learning.

Why is this important? We can now identify and work with teachers from

departments where there is evidence of significant ICT impact to find out HOW this has been accomplished


Dissemination /DevelopmentDissemination /Development

Dissemination Leaflet in BETT catalogue Reports provided to delegates at NAACE

conference (February) 2002 reports sent to all Primary and Secondary

schools in England, Northern Ireland, Scotland and Wales (June)

Reports available on Website (www.fischertrust.org)

Development 2002 surveys included with reports (June) Surveys available on-line Reports on overall trends in the data, links to

value-added analysis, case studies …


Opportunities and LimitationsOpportunities and Limitations

We now have: A database, currently with more than 25,000 evaluations by

3,500 teachers in 2,500 secondary departments A rapidly growing database of evaluations from primary

schools Evidence that use of Teacher Evaluation (together with Value-

Added Analysis) is a good mechanism for identifying examples of how the use of ICT can ‘make a difference’

What we don’t currently have: Time and resources to go beyond the publication of reports

and the development of a some case studies Therefore

Are there others interested in working with us to build upon this work, particularly in terms of finding key ‘success factors’ in High Impact ICT departments or schools

Any suggestions – Email ( [email protected] )

becta ict research conference – june 2002 intro survey details secondary surveys conducted july...

Documents