the evaluation of introduction to middle school journalism

8/4/2019 The Evaluation of Introduction to Middle School Journalism

1/21

The Evaluation of Introduction to Middle School Journalism

Shanda Veatch

Dept of Ed Tech, 512 Boise State University

Evaluation Criteria Explanation Data Sources

Effectiveness:

Mastery of WBI goals

Determine if students are

able to produce a school

newspaper

Student hascontributed an article

to the school

publication

Student has correctlyformatted a page for

the newspaper

Practice exercises inWBI

Appeal:

Gain and maintain learner

attention and usability

Review graphics,video and text for

interest

Review content forinterest

Review technologyfor navigability

Student reflection onengagement level of

graphics, video and

text

Instructor reflectionabout delivery

system

Alignment withEssential Academic

Learning Standards

Efficiency:

Instruction delivered in a

timely manner

Investigate time required for

curriculum delivery

Document instructorprep and delivery

time

Document studentWBI activity time

Evaluation Criteria and Categories Questions

Effectiveness

Goals Is the information accurate? Are the goals and objectives clear? Are goals and objectives achievable? Are the goals and content

appropriate for the method of


2/21

delivery

Content Is the information complete, coveringthe content properly?

Is there a match among content,objectives, activities, and assessment

tools? Are reference citations provided? Do the instructional activities reflect

current practices in Journalism?

Do the activities promote learning?Technology Do the technology applications

function properly?

Were materials easy to access bystudents?

Are copyright and intellectualproperty not violated?

Message Design Do supporting graphics and featuresenhance the learning without

distracting?

Is appropriate voice used inexpressing the content to the

learners?

Was humor used appropriately? Are directions clear? Was the time frame of the course

appropriate?

Appeal

Goals Are goals relevant to learners?Content Is the content interesting and

challenging?

Technology Are typographical, spelling andgrammatical errors distracting?

Are there coding errors? Is the code written in a user-friendly

protocol?

Is navigation easy?Message Design Is the media pleasing?

Is the vocabulary level and toneappropriate for the content and

middle school learners?

Are the screens uncluttered and withplenty of white space?


3/21

Is the color, typeface, emphasis usedappropriately and to enhance

learning?

Do supporting graphics and featuresenhance learning without

distracting? Do the graphic devices function

properly?

Are the graphics, animations, orsound clear?

Does it have good navigationaldesign? Is the meaning of the icons

clear and easy to use?

Are the screen layouts appropriatemiddle school Journalism goals?

Efficiency

Goals Are the goals stated clearly andconcisely?

Is the purpose stated clearly andconcisely?

Is there congruence between theinstructional goals and content?

Content Is the content information clearlyand concisely presented?

Is it appropriate to middle schoolJournalism?

Is it timely, up to date?Technology Is access to the instructor or other

learners provided?

Is the website structuredappropriately?

Do technology applications functioneasily and efficiently?

Message Design Is the organization and structure ofthe message coherent?

Are the titles and subtitles used toorganize the content?

Is the time frame for the curriculumappropriate for middle school?

Introduction to Middle School Journalism Stakeholders

Primary Stakeholders


4/21

Instructor: Shanda Veatch. She is responsible for delivery of instruction andcoordination for the publication of the school newspaper.

Designer: Shanda Veatch. Not only is she responsible for instructional delivery, butshe is in charge of designing Introduction to Middle School Journalism and

developing the evaluation plans.

Students: The students at Gaiser Middle School will directly impact the WBIthrough the piloting process.

Secondary Stakeholders

Michael Lane. He is the principal of Gaiser Middle School, and Shanda Veatchsdirect supervisor. He will be providing feedback to Shanda about the quality of the

publication and the program.

Rose Crismon. She is the IT specialist at Gaiser Middle School. She will be workingwith Shanda Veatch to ensure student access to all elements and modules of the

WBI.

Gaiser Middle School community. This is a project based course, and the finalproduct is presented to the students, staff and extended GMS community. The

quality of this publication will represent Gaiser to this community.

Materials to be Examined

Design Plans Objectives Assessment material Curriculum organization

Delivery of instruction Student engagement level

Prototype and Website Navigation Interface with VPS server Graphics WBI prototypes

Evaluators and Reviewers

Evaluator/Designer/Instructor: Shanda Veatch is responsible for creating the WBI and

delivering the curriculum to students at Gaiser Middle School. She has taken many coursesin Education Technology, and created multiple assessments, both formative and

summative. She will be analyzing the data created curing the pilot of this curriculum for

dissemination by her instructor at Boise State University and her supervisor at Gaiser

Middle School


5/21

Expert Reviewer (Instructional Design): K Dianne Hall, Ed.D. Instructional Technology,

Texas Tech University. Professor Hall is an adjunct professor at Boise State University and,

as of the writing of this design document, is the instructor of Online Course Design for the

Masters in Education Technology program at Boise State.

End User Reviewer: Students at Gaiser Middle School who are participating in the

Journalism program during the 2010-2011 school year will participate in the piloting of

this curriculum. They will be providing feedback through formative evaluations as well as

providing regular feedback during the piloting process.

Expert Reviewer (Subject Matter): Rose Crismon is currently heading information

technology department at Gaiser Middle School. She is certified CCNA and provides staff at

both Gaiser and a local elementary school with all technology support. She is also network

administrator for both schools.

Evaluation Criteria and Categories Question Methods and Tools

Effectiveness

Goals Is the informationaccurate?

Are the goals andobjectives clear?

Are goals and objectivesachievable?

Are the goals andcontent appropriate for

the method of delivery

Expert, Instructor with

checklists, End user

review and survey,Extant data (practice

exercises, final

publication)

Content Is the informationcomplete, covering the

content properly?

Is there a match amongcontent, objectives,activities, and

assessment tools?

Are reference citationsprovided?

Do the instructionalactivities reflect current

practices in Journalism?

Expert, Instructor with

checklists, End user

review and survey,

Extant data (practice

exercises, final

publication)


6/21

Do the activities promotelearning?

Technology Do the technologyapplications function

properly?

Were materials easy toaccess by students?

Are copyright andintellectual property not

violated?

Expert (ID, SME,

instructor with

checklists), End-user

review/interview, and

observation

Message Design Do supporting graphicsand features enhance the

learning without

distracting?

Is appropriate voice usedin expressing the content

to the learners?

Was humor usedappropriately?

Are directions clear? Was the time frame of

the course appropriate?

Expert (ID, instructor

with checklists), End-

user review, observation

Appeal

Goals Are goals relevant tolearners?

Expert (ID, SME), End-

user review

Content Is the content interestingand challenging?

Expert (instructor with

checklists), Observation

Technology Are typographical,spelling and

grammatical errors

distracting?

Are there coding errors? Is the code written in a

user-friendly protocol?

Is navigation easy?

Expert (ID, Instructor

with checklists), End-

user observation, End-

user formative

checklists

Message Design Is the media pleasing? Is the vocabulary level

and tone appropriate for

the content and middle

school learners?

Are the screens

Expert (ID, SME), End-

user survey, observation


7/21

uncluttered and with

plenty of white space?

Is the color, typeface,emphasis used

appropriately and to

enhance learning? Do supporting graphics

and features enhance

learning without

distracting?

Do the graphic devicesfunction properly?

Are the graphics,animations, or sound

clear?

Does it have goodnavigational design? Isthe meaning of the icons

clear and easy to use?

Are the screen layoutsappropriate middle

school Journalism goals?

Efficiency

Goals Are the goals statedclearly and concisely?

Is the purpose statedclearly and concisely? Is there congruence

between the

instructional goals and

content?

Expert (SME, ID,

Instructor with

checklists), End-userobservation and

interview

Content Is the contentinformation clearly and

concisely presented?

Is it appropriate tomiddle school

Journalism? Is it timely, up to date?

Expert (ID, Instructor),

End-user observation

and interview

Technology Is access to theinstructor or other

learners provided?

Is the website structuredappropriately?

Expert (ID, SME and

instructor with


observation, Extant data


8/21

Do technologyapplications function

easily and efficiently?

Message Design Is the organization andstructure of the message

coherent? Are the titles and

subtitles used effectively

Is the time frame for thecurriculum appropriate

for middle school?

Expert (SME, ID,

Instructor with


observation

Introduction to Middle School Journalism Evaluation Methods and Schedule

This program will be field-tested and formative assessments will be conducted throughout

the design, implementation and review faze of this project. Review will be ongoing by the

subject matter expert, the instructional designer, the instructor as well as the students who

are testing this curriculum. Adjustments will be made as a result of their feedback at each

stage of the process. There will be between 27-30 students participating in this program,

and, as they are currently taking middle school Journalism at Gaiser Middle School, their

participation will be compulsory. Permission for participation will not be necessary, as the

individual participants will not be named.

Modules will be delivered as they are created, in order to be modified to meet the needs of

the learners. Delivery of the curriculum will not be in chronological order in this field test,

but each module will be examined by the above listed experts as well as the end-users for

logical flow from one to the next. Due to time constraints, the field testing timeline will not

encompass an entire trimester. Depending on the feedback, this may have a substantial

impact the final project length.

Ongoing Decisions Made while Developing and revising the WBI Design and

Prototype

With each step of the implementation of the Introduction to Middle School Journalism field

test, data will be collected from the end-users and curriculum will be modified to reflect the

changes that need to be made as a result. Shanda will share all reported technical data

incongruencies with the SME, Rose Crismon, in order to evaluate the best recourse for

reconciliation of the data with the needs of the learners.


9/21

Students will be producing a full publication as a result of this WBI. The final project will

be analyzed by the ID and the SME as well as the administrator at Gaiser Middle School,

Michael Lane, and reviewed for attainment of learning goals and curriculum alignment with

state standards. Inferential data, as it relates to end-user satisfaction will be shared with

the team. The team will determine how that will impact full implementation the final

product.

Preliminary Planning for Summative Evaluation

Evaluation Criteria Main Questions Data Sources

Effectiveness Did the final projectmeet with the

instructional goals

and objectives?

Did the WBI contentand assessment align

with state standards?

Individual studentactivities

Goal review byinstructional ,

administrative andexpert team

Efficiency What was the averagelength of time

students were

engaged in each

module of the course?

What was the totaltime invested in the

classroom for the

WBI project as awhole?

Documentedclassroom time per

module

Documented wholeclass instructional

time

Appeal Will this programstand alone without

instructional

assistance by the ID?

Can this WBI bereplicated in other

classrooms or as an

entirely virtual

presentation?

Delivery of WBI tooffsite instructor for

implementation in a

separate classroom

and review

Review by instructorsin virtual learning

environment

The time frame for this project extends beyond the trial period presented so far in this

design document. The field testing will take place during the 2010-2011 school year. In

order to meet the needs of the Instructional Designer, Shanda Veatch, who is designing this

curriculum for her Masters at Boise State University, initial design must be completed by


10/21

December of 2010. A preliminary summative evaluation will not be complete until the end

of the school year.

Since there has never been a WBI for Journalism at Gaiser Middle School, there is no

baseline data from which to draw any comparative data. There have been other curriculum

delivery methods, and the comparative data will be based on the overall success of student

scores as a whole drawn from the past three years. We will be looking for improvement in

overall classroom grade point averages as well as decreased failure rates and increased

student engagement among Journalism students at Gaiser Middle School.

Shanda Veatch, as the instructor, will oversee the field testing of this curriculum, and as a

result will utilize observation, questionnaires, surveys and interviews along with the

assessment data to determine the effectiveness of this WBI. Formative evaluation data will

be analyzed in order to improve the effectiveness of the delivery systems as well as the

content. The summative evaluation will be provided in a preliminary report to the GaiserMiddle School administrative team at the end of the school year.

The process will be undertaken again the following fall of 2011, allowing for a second

summative evaluation to be performed before the end of that year. This evaluation will be

provided to the stakeholders at the end of that school year as a formal report, whereupon

the curriculum will be delivered to another teacher for instruction during the 2012-2013

school year.

At the conclusion of this third year, the findings of the new instructor will be provided to

the Instructional Designer, and a final summative evaluation report will be drafted for

delivery to the head of the Careers in Technical Education at Vancouver Public Schools. It

is with hope that this curriculum will be available for regular delivery by middle school

teachers in the Vancouver School District at that point.

Pre-Planning Activities

WBI Project Timelines

Tasks Persons

Responsible

Week

1

Week

2

Week

3

Week

4

Week

5

Week

6

Week

7

Conduct

preplanning

activities

Veatch Projected

Actual


11/21

Write

Objectives

Veatch Projected

Actual

Write

assessment

items and

tools

Veatch Projected

Actual

Evaluate

objectives

and

assessment

items

Veatch Projected

Actual

Cluster and

sequence

objectives

Veatch Projected

Actual

Create WBI

strategy

worksheet

Veatch Projected

Actual

Identify

media

Veatch Projected

Actual

Evaluate

instructionalstrategies

and media

selections

Veatch Projected

Actual

Flowchart

and

Veatch Projected


12/21

storyboard

lesson

Actual

Week8

Week9

Week10

Week11

Week12

Flowchart

and

storyboard

lesson

(Cont)

Veatch Projected

Actual

Design

website

Veatch,

Crismon

Projected

Actual

Convert

storyboard

to web

pages

Veatch Projected

Actual

Conduct

evaluation

(test

website and

instructional

quality of

WBI,

troubleshoot

technology,

test

Veatch,

Crismon

Projected

Actual

Final Project

Due

Veatch Projected

Actual


13/21

Task-Objective-assessment Item Blueprint

For Introduction to Middle School Journalism

Learning Task

Item and Number

Objective Learning Outcome

Students will create

a newspaper for

publication

Given the articles and

accompanying photographs

and graphics, students will

use InDesign to produce a

newspaper for school-wide

distribution

Intellectual skill:

Synthesis,

generalization

Is the paper

free of

grammatical

errors?

Are the

pictures clear?

Does each

picture have a

caption?

Is there a

headline and

byline for each

article?

Are the

articles

formatted

correctly?

Is there a

border on

every page?

1.0 Students will lay

out a page for the

newspaper

Students will use InDesign

to lay out a page for the

newspaper

Intellectual skill:

Synthesis, retrieval

Is the page

formatted

correctly?

Does each

article have a

Headline and

byline?

Does each

article have a

graphic or


14/21

photograph?

Does each

photograph

have a

caption?

1.1 Students will

select the articles

and pictures for

publication

Students will read and

analyze the articles

submitted for the

publication and choose the

ones for publication

Intellectual skill:

Evaluation,

reinforcement

Which articles

are formatted

correctly?

Which articles

are free of

grammatical

errors?

Which pictures

are clear?

Which articles

are most

relevant to the

current issue?

2.0 Students will

write a human

interest story

Having interviewed a

subject, students will write

a 200-250 word article

including direct and

indirect quotations, and

photographs of the subject

Intellectual skill:

Synthesis,responding

Is the article a

narrative, with

quotations

woven into the

text?

Are the

pictures clear

and easy to

view?

2.1 Students will

photograph their

subjects

Students will take several

photographs of their

subjects during the

interview

Intellectual skill:

Application,

responding

Is the subject

centered in the

frame?

Is the lighting

conducive to

publication?


15/21

Is the photo

clear and easy

to view?

2.2 Students will

compile a list of

quotes from their

interview

Students will record at least

10 quotes from the

interview

Intellectual skill:

Analysis, retrieval

Which

quotations

define the

personality?

Which

quotations

provide insight

into the

subject?

Whichquotations

help to further

the story?

2.3 Students will

conduct an

interview

Students will schedule and

arrange an interview place

and time

Intellectual skill:

Application,

responding

Did the

student

contact the

interviewee in

advance of the

interview toschedule a

time?

Was the

student

prepared with

questions,

camera and

writing tools

for conducting

the interview?

Did the

student check

the quotes

with the


16/21

interviewee

before

dismissing his

or herself?

Did theinterviewer

thank the

interviewee?

2.4 Students will

prepare a list of

questions for the

interviewee

Students will generate a list

of at least 20 questions that

they can ask the

interviewee

Intellectual skill:

Synthesis, retrieval

What was the

reason for

choosing this

subject?

Are thequestions

relevant to

telling a story?

Do the

questions elicit

extended

responses

from the

interviewee?

2.5 Students will

choose a subject to

interview

Students will choose an

interviewee and explain

their rational for choosing

this person

Intellectual skill:

Analysis, selective

perception

What makes

this person

relevant to

school culture?

What would

you like to

learn about

this person?

2.6 Students will

identify appropriate

interviewing

etiquette

Students will create an

interviewing etiquette

themed powerpoint and

conduct a mock interview

with a classmate,

Intellectual skill:

Application,

synthesis, semantic

encoding

How do you

set up an

interview?

How do you


17/21

demonstrating correct

interviewing etiquette

approach an

interviewee

when

beginning the

meeting?

How do you

ask for

clarification?

How to ask

follow up

questions?

How to check

forinaccuracies in

the

quotations?

How to close

the interview

3.0 Students will

write an editorial or

draw an editorialcartoon

Students will choose from

the editorial topics to write

an opinion piece on thissubject

Intellectual skill:

Synthesis,

responding

What about

this subject do

I agree with ordisagree with?

What is the

counter

argument?

What is my

solution to the

problem?

3.1 Students will list

appropriate

editorial topics for

the newspaper

Students will discuss

relevant school-related

topics, and choose at least

three for the publication

Intellectual skill:

Evaluation, retrieval

What are some

school-related

debates that

are happening

on campus and

in the


18/21

classrooms?

3.2 Students will

identify the

structure used in

opinion writing

Students will be able to

identify the structure of

opinion writing (thesis

statement, argument,

counter argument, solution,

restatement of the thesis)

Intellectual skill:

Knowledge,

responding

How do I

organize my

writing for an

opinion

article?

3.3 Students will

learn the

characteristics of

opinion writing

Given two pieces of writing,

one a news story and one an

editorial, students will be

able to discern which is an

editorial with 100%

accuracy

Intellectual skill:

Application,

selective perception

What is the

difference

between a

news article

and an opinion

article?

4.0 Students will

write a feature

article for the

newspaper

Students will organize their

ideas into a short 200-250

word feature article on one

of the selected topics

Intellectual skill:

Synthesis,

generalization

Why would my

audience find

this subject

interesting?

What can I do

to make them

want to read

on?

4.1 Students will

select feature topics

for the newspaper

Students will brainstorm a

list of school-related topics

to write about

Intellectual skill:

Evaluation,

responding

What are some

successes that

have been

noted on

campus?

What are some

changes we

have

witnessed?

What makes

us unique as a

community?

4.2 Students will When presented with two Intellectual skill: Based on my


19/21

identify the

differences between

news writing and

feature writing

articles, one a news article

and one a feature story,

students will identify the

feature story with 100%

accuracy

Application,

semantic encoding

prior

experience,

what tools do I

now have to

help me

understand

the differences

between these

two styles of

writing?

4.3 Students will

identify the

characteristics of

feature writing

Students will be able to

explain the essential

elements of feature writing

(narrative/expository,catchy opening, sequences

information, reaches a

conclusion) with 100%

accuracy

Intellectual skill:

Knowledge,

semantic encoding

How is a

feature story

organized?

What makes it

unique to

journalistic

writing?

5.0 Students will

write a news story

for the newspaper

Given the topics generated,

students will write a short

news story employing the

5Ws and the H

Intellectual skill:

Synthesis,

generalization

Did I identify

the who, what,

where and

when at the

beginning ofthe article?

Did I discuss

the why and

the how at the

conclusion of

the article?

5.1 Students will

choose newsworthytopics for the

newspaper

Students will generate a list

of at least 4 school-relatednews stories for publication

in the paper

Intellectual skill:

Evaluation, retrieval

What are some

things thathave occurred

on campus

that may not

be widely

known?


20/21

What athletic

events, social

events,

classroom

events or

extracurricular

events have

been reported

to you lately?

5.2 Students will

identify the features

of the inverted

pyramid structure of

a news story

Students will identify the 4

Ws as the pyramid base and

the Why and How as the tip

on a graphic organizer

Intellectual skill:

Application,


What is the

inverted

pyramid and

how will it

help me towrite a story?

5.3 Students will

identify the essential

elements of a news

story

Students will Students will

be able to answer the 5 Ws

and the H after reading a

short news story

Intellectual skill:

Application,


What

happened to

whom? Where

did happen?

When did it

happen?

Why did it

happen and

how did it

happen?

6.0 Students will

determine the

format of the school

newspaper

Student will determine the

order of sections for the

school newspaper (ie: news,

feature, human interest,

sports, editorial/opinion,

games/puzzles/art/poetry)

Intellectual skill:

Evaluation,

generalization

What makes a

newspaper

readable?

How will our

paper entice

readers to

read on?

6.1 Students will

identify different

sections in a

Students will create a

virtual collage using

InDesign in order to

identify the different

Intellectual skill:

Application,

synthesis, semantic

What sections

can we find in

the local


21/21

newspaper sections of a newspaper

(news, sports, feature,

opinion, lifestyle)

encoding paper?

What sections

can we find in

our school

newspaper?

6.2 Students will

identify vocabulary

related to

newspapers

Students will identify the

parts of a newspaper with

at 100% mastery.

(Masthead, headline, byline,

caption, cutline)

Intellectual skill:

Knowledge,

semantic encoding

What are the

parts of the

newspaper?

Where will I

find the

masthead,

headline,

byline, captionand cutline in

a traditional

newspaper?

6.3 Students will

identify the

purposes of a

newspaper

As a class, students will

create a list of things that

can be found in a

newspaper

Intellectual skill:

Application,

reception

Why do you

read the

newspaper?

Why do your

parents readthe paper?

What other

reasons can

you

Introduction to Middle School Journalism Objective Clusters

1. Create your page! (1.0, 1.1)2. Writing a profile (2.0- 2.6)3. Whats your opinion? (3.0- 3.3)4. The feature article (4.0- 4.3)5. Time for the news (5.0-5.3)6. Whats a newspaper? (6.0-6.3)

the evaluation of introduction to middle school journalism

Documents