Monitoring and Evaluation of ICT Use in Rural Schools

Ok-choon Park

Global Symposium on ICT in Education – Measuring Impact: Monitoring and Evaluation of ICT in Education Initiatives

November 9 – 11, 2009Seoul, Korea

1

Should monitoring and evaluation of ICT use in rural areas be different from those in other areas?

Probably little for logistics

Important difference should be the purpose of ICT use.

Why ?

2

But not for basic procedures and methods

Rural schools have different contextual characteristics and needs.

• Geographically isolated.

• Smaller student populations, smaller class sizes, and more combined classes

• Fewer well qualified teachers

• Lower SES of parents

• Fewer opportunities for supplemental education services

• Fewer resources and supports inside and outside schools

What are important characteristics of rural schools?

Fewer Educational Opportunities

3

Lower Educational Quality

Fewer Educational Opportunities

• Fewer advanced courses; fewer enrichment classes; fewer remedial classes; fewer elective courses

• Fewer options for extra curriculum activities

• Fewer internship opportunities

• Fewer field trip options

• Fewer mentoring opportunities

• Fewer options for supplemental education

• Others

4

Lower Educational Quality

• Fewer opportunities Lower quality

• Less experienced and less qualified teachers

• Fewer resources

• Less stimulating interactions with peers

• Less parental assistance

• Others

5

ICT for Expanding Educational Opportunities

• Distance learning, virtual classrooms, and virtual schools

• Access to web-based resources

• On-line library

• On-line mentoring

• Virtual field trips

• Cooperative learning with students in other schools

• Others

6

• On-line courses and supplemental instruction materials

• ICT-based learning tools and resources

• Virtual laboratory experiments, field trips, and others

• On-line teaching aids for teachers

• Data management tools for teachers Data-based teaching

• Administrative and management tools for teachers More teaching time

• On-line cooperative learning environments

• On-line interactions between teachers, students and parents

• Others

Enhancing Educational Quality

7

Technology is a double-edged sword!

• Technology can expand educational opportunities and enhance educational quality for all children.

• But, technology creates digital gaps between rural and other poor schools and wealthy schools.

Digital gaps widen:

• Closing the gaps is a challenging responsibility for policy makers!

8

Educational opportunity and quality gaps

Learning achievement gaps

Social and economic gaps

Initiative Examples of the US Government for Narrowing the Gaps

• Early initiatives for educational technology

- PLATO system – hardware and software development - Computer Curriculum Corporation – Courseware development

• Star Schools – Distance learning

• E-Rate – ICT infrastructure building

• 21st Community Learning Centers – After-school learning

• Many other grants for educational technology

These and other government initiatives stimulated private investments in educational technology and its use at schools and homes!

9

Current Access to ICT in the USA Schools • Near universal access to the Internet for all schools and classrooms

• One computer for every 3 or fewer students

But, digital gaps and educational opportunity and quality Gaps are still there!

• Limited access to high-speed broadband internet

• Fewer computers for students and teachers

• Less technical support

• More students without computers or Internet connections at home

• Fewer resources

• Less parental support, and others

10

• Laptop loan programs – Pennsylvania example

- A laptop for every high school student

- On-site technical support (1/2 time technician)

- Teacher technical training – 30 hours

Recent Trends for ICT Use in the USA

11

• Virtual schools and classrooms

- Increasing number of state-wide and independent virtual schools and classrooms

- Widely different between states, districts, and schools: Course offerings and requirements

• Digital textbook programs

Needs of Systematic Monitoring and Evaluation for ICT Use

• How much do we know about current ICT access and use in schools and homes?

• Do we know what impacts ICT use has had for schools, teachers, and children?

• How do we know what supports schools, teachers and students need for effective use of ICT?

• How can we provide systematic guidance for teachers, parents and students to choose and use ICT programs and features?

• Do we have clear ideas how to improve ICT uses in schools and homes?

• How do we know which programs are good or bad?

12

Two Types of Evaluation for ICT Use

• Descriptive evaluation of ICT Use (e.g., NCES survey)

- Information gathering for ICT access and use

- Surveys, observations, interviews, etc.

13

• Impact Evaluation of ITC Use (e.g., IES RCT studies)

- Investigation of impacts of specific ICT features and programs on student learning

- Experiments, quasi-experiments, regression discontinuity analysis, etc.

• Access at schools, classrooms and homes for teachers and students

- Internet access - Computer availability - Technical support

• Use of ICT features and programs

- Instructional and instructional support use - Administrative and management use - Other uses – communication

• Extent of ICT use – frequencies and lengths

• Barriers for ICT use

Descriptive Evaluation for ICT Access and Use

14

Longitudinal Database of Descriptive Evaluation Data

• Trend analysis of ICT access and use

• Correlational studies between ICT use and other variables

- Teacher instructional methods

- Student study patterns

- Interaction patterns among teachers, students and parents

- Student learning achievements

- Student attitudes

- Others

15

• Need of scientific evidence-based information for the users and policy makers

• Limited value of past evaluation study results

- Laboratory-based experiments

- Meta-analysis results

- Quasi-experiments and descriptive studies

- Correlational, descriptive, and case studies

Needs for Impact Evaluation of ITC Use

16

• Selection of specific ICT use or programs for evaluation: Justifications

- Theory-based - Demonstrated efficacy

- Practical justification – wide use

- Availability of implementation requirements and guidance

Methodological Considerations for Impact Evaluation of ICT Use

17

• Selection of evaluator

- Objective third party: Firewall between developer/implementer and evaluator

- Experience and technical competence

• Selection of evaluation method

- Randomized control study

- Quasi-experiment

- Regression discontinuity analysis

18

• Selection of samples

- Students

- Teachers/classrooms

- Schools

• Determination of generalization level of the results

- Fixed effect

- Random effect

- Determination of sample size: Power analysis

19

• Selection of outcome measures

- Terminal outcome measures – student achievements Tailored measures Standard assessments

- Intermediate outcome measures Teacher knowledge Instructional methods Student attitudes Others

• Selection of study scope

- Data collection points

- Number of cohorts

- Selection of implementation feasibility data: surveys, observations, interviews, etc.

20

• Development of analysis plan

- Analysis models

- Exploratory data analysis; Sensitivity analysis; subgroup analysis

- Missing data management

- Multiple comparison

• Recruitment of samples

• Assignment of samples to treatment and control conditions

21

• Implementation: as required by developer

- Teacher training

- Classroom implementation

- Technical supports

• Data analysis and report generation

- Confirmatory analysis – Summative report

- Exploratory analysis – Formative report

- Report with or without the results and interpretation of exploratory data