part 2: evaluating your program

Part 2: Evaluating your program

Part 2: Evaluating your program

You can download this presentation at:

http://faculty.smcm.edu/acjohnson/PREP/

You can download this presentation at:

http://faculty.smcm.edu/acjohnson/PREP/

Evaluating your programEvaluating your program

What are your objectives for your program?

What data would let you know you’re meeting those objectives?

What data would convince your administration to keep funding the program?

What are your objectives for your program?

What data would let you know you’re meeting those objectives?

What data would convince your administration to keep funding the program?


Informal assessments: To let you know that the program is working; to fine-tune it as you go along

Observations of student progress, conversations with students, informal surveys

Informal assessments: To let you know that the program is working; to fine-tune it as you go along

Observations of student progress, conversations with students, informal surveys


Formal assessments Of the first year Longer-term

Formal assessments Of the first year Longer-term


The basics: Comparison groups Independent variables Dependent variables

The basics: Comparison groups Independent variables Dependent variables

Comparison groupsComparison groups

Historical: ESP participants vs similar students before ESP

Comparable: ESP participants vs similar students not in ESP

To the norm: ESP participants vs all non-participants

To decliners: People who rejected an invitation to ESP

Historical: ESP participants vs similar students before ESP

Comparable: ESP participants vs similar students not in ESP

To the norm: ESP participants vs all non-participants

To decliners: People who rejected an invitation to ESP

Independent variablesIndependent variables

ESP participation Race Gender Academic preparation (SAT scores;

CCI pre-test) Financial need Motivation

ESP participation Race Gender Academic preparation (SAT scores;

CCI pre-test) Financial need Motivation

Dependent variablesDependent variables

CCI post-test scores CCI growth scores Calc grades

Raw data, % receiving A or B, % failing

Enrollment/grades in Calc II Declaring SEM major Graduating at all Graduating with SEM major

CCI post-test scores CCI growth scores Calc grades


Enrollment/grades in Calc II Declaring SEM major Graduating at all Graduating with SEM major

Analyzing the dataAnalyzing the data

Descriptive statistics: Simply compare the performances of the relevant groups

Are differences in grades or scores significant? Independent-samples t-tests

Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square

Descriptive statistics: Simply compare the performances of the relevant groups

Are differences in grades or scores significant? Independent-samples t-tests

Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square


Controlling for preparation: Divide data into groups according to some measure of preparation

Controlling for preparation: Divide data into groups according to some measure of preparation


Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest

For continuous dependent variable: OLS; for binary: Logistic regression

Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest

For continuous dependent variable: OLS; for binary: Logistic regression

Calculus concept inventory

Calculus concept inventory

Pros This is the gold standard--did the students learn

calculus? Did they learn more than other students? This approach--at least the descriptive stats--can be

used for small n Cons

Limited number of test items--test might not be reliable or valid enough for your comfort

Requires access to all calculus students, not just ESP students

Pros This is the gold standard--did the students learn

calculus? Did they learn more than other students? This approach--at least the descriptive stats--can be

used for small n Cons

Limited number of test items--test might not be reliable or valid enough for your comfort

Requires access to all calculus students, not just ESP students

Calc grades, SAT & GPA data

Calc grades, SAT & GPA data

Pros: Lets you control for preparation Administrators like statistical analyses

Cons: Someone has to like stats--might need SPSS You have to find someone in institutional research to

let you have the data Requires a substantial n

Pros: Lets you control for preparation Administrators like statistical analyses

Cons: Someone has to like stats--might need SPSS You have to find someone in institutional research to

let you have the data Requires a substantial n

ExamplesExamples

Fullilove & Treisman, 1990 Comparison groups:

Historical--pre-MWP African Americans African American accepters & decliners

Preparation measures: Special admission? Math SAT scores

Dependent variables: Calc performance, graduation

Fullilove & Treisman, 1990 Comparison groups:

Historical--pre-MWP African Americans African American accepters & decliners

Preparation measures: Special admission? Math SAT scores

Dependent variables: Calc performance, graduation

ExamplesExamples

Johnson, 2007a Comparison groups (all with 1st major

in science): White/Asian; Black/Latino/American

Indian

Independent variables: Financial need, predicted GPA

Dependent variables: Graduation with science/math major, grad GPA

Johnson, 2007a Comparison groups (all with 1st major

in science): White/Asian; Black/Latino/American

Indian

Independent variables: Financial need, predicted GPA


Expanding your programExpanding your program

Evidence that matriculation-to-graduation programs produce even bigger benefits:

Johnson (2007a) Maton, Hrabowski & Schmitt (2000) Maton & Hrabowski (2004) Gándara (1999)

Evidence that matriculation-to-graduation programs produce even bigger benefits:

Johnson (2007a) Maton, Hrabowski & Schmitt (2000) Maton & Hrabowski (2004) Gándara (1999)

Evaluating your programWhat are your objectives for your program?What data would let you know you’re meeting those objectives?What data would convince your administration to keep funding the program?

Evaluating your programInformal assessments: To let you know that the program is working; to fine-tune it as you go alongObservations of student progress, conversations with students, informal surveys

Evaluating your programFormal assessments

Of the first yearLonger-term

Evaluating your programThe basics:

Comparison groupsIndependent variablesDependent variables

Comparison groupsHistorical: ESP participants vs similar students before ESPComparable: ESP participants vs similar students not in ESPTo the norm: ESP participants vs all non-participantsTo decliners: People who rejected an invitation to ESP

Independent variablesESP participationRaceGenderAcademic preparation (SAT scores; CCI pre-test)Financial needMotivation

Dependent variablesCCI post-test scoresCCI growth scoresCalc grades


Enrollment/grades in Calc IIDeclaring SEM majorGraduating at allGraduating with SEM major

Analyzing the dataDescriptive statistics: Simply compare the performances of the relevant groupsAre differences in grades or scores significant? Independent-samples t-testsAre differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square

Analyzing the dataControlling for preparation: Divide data into groups according to some measure of preparation

Analyzing the dataControlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interestFor continuous dependent variable: OLS; for binary: Logistic regression

Calculus concept inventoryPros

This is the gold standard--did the students learn calculus? Did they learn more than other students?This approach--at least the descriptive stats--can be used for small n

ConsLimited number of test items--test might not be reliable or valid enough for your comfortRequires access to all calculus students, not just ESP students

Calc grades, SAT & GPA dataPros:

Lets you control for preparationAdministrators like statistical analyses

Cons:Someone has to like stats--might need SPSSYou have to find someone in institutional research to let you have the dataRequires a substantial n

ExamplesFullilove & Treisman, 1990

Comparison groups: Historical--pre-MWP African AmericansAfrican American accepters & decliners

Preparation measures:Special admission? Math SAT scores

Dependent variables:Calc performance, graduation

ExamplesJohnson, 2007a

Comparison groups (all with 1st major in science):White/Asian; Black/Latino/American Indian

Independent variables:Financial need, predicted GPA


Expanding your programEvidence that matriculation-to-graduation programs produce even bigger benefits:Johnson (2007a)Maton, Hrabowski & Schmitt (2000)Maton & Hrabowski (2004)Gándara (1999)

part 2: evaluating your program

Documents