transition assessment and goal generator (tagg): a new on-line assessment

JIM MARTIN & AMBER MCCONNELL

UNIVERSITY OF OKLAHOMADEPT OF ED PSYCHOLOGY

ZARROW CENTER

Transition Assessment and Goal Generator (TAGG): A New On-Line Assessment

Quality Transition Education Produces Greater Outcomes

Transition Education Begins With Transition Assessment

Illustration of two high school graduates. One on large horse and student suited in armor. This student had quality transition education while in school. The other student in on a stick-horse going in the wrong direction. He did not have quality transition education.

Agenda

Purpose of Transition AssessmentUse results to build postsecondary

and annual transition goalsQuestions to Ask to Pick a Transition

AssessmentIs this a good transition assessment

to use?Importance of Annual Transition

GoalsThe TAGG

Purpose of Transition Assessments

Questions Students Need to Use Results to Build Postsecondary Goals

Postsecondary Goal QuestionsUpon graduation from high school:Where do I want to live?Where do I want to work?Where do I want to learn to do the

job I want?

Questions for Annual Transition Goals

What do I need to learn now:To live where I want after graduating

from high school?To do the job I want after graduating

from high school?To learn where I want after

graduating from high school?

A Few Questions to Ask to Help Pick a Transition Assessment

Validity Evidence

Test developer needs to provide basic validity evidence

User needs to examine evidence and assessment to determine if it is appropriate for students and intended purpose

Importance Factor

The more important decision, the more validity results need to be

How valid and reliable are the results used to make transition planning decisions?

Often not easy to find outDuties . . .

Sources of Validity Evidence

1. Item Development2. Items Structured and Organized3. Internal Reliability4. Stability of Scores (Test-Retest)5. Fairness Validity6. Match with other similar

assessments: Concurrent Validity7. Purpose: Does it fit for your uses?

Development

How were the items developed?Did the authors just sit down and make them

up?Did the authors ask knowledgeable sources?Did the authors use findings from research

with students with disabilities?Are they based on what students need?Are items developed, then clustered into

similar groups?Are cluster definitions defined first then

items developed?

More Item Questions

How are the items structured and organized?Based on common sense categories?Based upon statistical analyses of

user responses to verify theory behind assessment?

ReliabilityDoes evidence of acceptable internal consistency scores exist?Cronbach’s alpha (.7 or higher)

Does evidence exists that when same users first completed the tool, then completed it again several weeks later, the results are similar?Does evidence of good test-retest reliability exist?

Validity

If the assessment fair? Does evidence exist that no major differences exist acrossRaceEthnic groupsGenderSES

Does evidence exist that the assessment results match those from another assessment that examines the same concept?Correlation between Weschler IQ and

Standford Binet IQ results

Importance Factor

The more important decision, the more validity evidence needed

Often not easy to discover validity evidence

Determining validity is up to the test developers and users

Example of What Can Be Learned

The Career Clusters Interest SurveyFreeSupposed to Identify Interests by the

16 career clustersEasy to use and scoreMany educators use it So, study by Prime and Tracey (2010)

. . . . ..

Study Results

The CCIS is reliable, but not validIt measures well time after time, but

The results are very limited in terms of what is being measured

Doesn’t match-up well with another assessment that measures the same concepts

“Its usage could provide a severely restricted set of options . . .”

Process & Decisions None Low Moderate High

Awareness X

Exploration X

PLEP X X

Strengths X X

Needs/Limits X X

Postsecondary Goals X X

Annual Transition Goals X X

Acceptable Degree of ValidityChart of IEP stages by the level of validity evidence – none, low, moderate, or high

TRANSITION ASSESSMENT AND GOAL GENERATOR

DEVELOPED WITH A GRANT FROM THE NATIONAL CENTER FOR

SPECIAL EDUCATION RESEARCH AND ZARROW CENTER FUNDS

The TAGG

TAGG Overview

An on-line transition assessment for secondary-aged students with disabilities, their educators, and parents designed to provide non-academic annual transition goals to enable students to learn skills to increase likelihood of employment and further education

TAGG Will Provide

The TAGG will provide A graphic profileWritten summary to copy and paste

into PLEP section of the IEPListing of relative strengthsListing of relative needsAnnual transition goal suggestions

matched to needs

Annual Transition Goal Questions the TAGG Can Answer

What do I need to learn now:To do the job I want after graduating

from high school?To learn where I want to learn after graduating from high school?

How Were the TAGG Items Developed?

The research team found research studies that identified behaviors of former high school students with disabilities associated with post high school employment or further education

The research team developed 10 construct definitions

Items developed from constructsOver three years, almost 1,500 students, 130

educators, and 800 parents from across the country completed the TAGG.

Initial Structure: Ten Initial Constructs

Knowledge of strengths and limitations

Actions related to strengths and limitations

Disability awareness

EmploymentGoal setting and

attainment

PersistenceProactive

involvementSelf-advocacySupportsUtilization of

resources

Sample Construct Definition

Disability AwarenessSuccessful individuals know they have a disability and can express needs to others in a non-stigmatizing manner. Individuals demonstrate knowledge of the disability and can express positive and negative aspects. They express information such as how the disability affects life and what supports are needed and legally allowed to compensate in various situations. The student needs to be able to place the disability within the context of his or her life and is not defined by the disability.

Year 1 and Efforts to Build TAGG Structure

Users completed the TAGGApplied various factor analyses

statisticsWent from 10 constructs to 8Went from 75 items to 34Construct structure confirmed by

second study with year 2 data

Professional and Family Constructs After Year 1 Analysis

Stayed1.Strengths and

Limitation

2.Disability Awareness

3.Persistence

4.Interacting with Others

5.Goal Setting and Attainment

6.Employment

7.Student Involvement in IEP

8.Support Community

Dropped1.Action Related to

Strengths and Limitation

2.Utilization of Resources

TAGG-P: (c2=1043.62, df=499, RMSEA=.058, CFI=.92, TLI=.91,

RMSR=.0597)TAGG-F: (c2=862.74, df=499,

RMSEA=.057, CFI=.91, TLI=.90, RMSR=.058)

Student Version Constructs After CFA

After CFA Constructs

1.Strengths and Limitations & Support Community

2.Disability Awareness

3.Persistence

4.Student Involvement in IEP

5.Interacting with Others

6.Goal Setting and Attainment

7.Employment

Dropped Construct

1.Action Related to Strengths and Limitation

2.Utilization of Resources

Combined Constructs

3.Strengths and Limitations

4.Support Community

TAGG-S: (c2=819.00, df=505, RMSEA=.047, CFI=.89, TLI=.88, RMSR=.064)

Internal Reliability

Does evidence exist that across the TAGG various items respond in the same way?Each TAGG version has great overall

internal consistency and satisfactory subscale consistency (ranging from α =. 89 to α =. 95)

Test-Retest Reliability

Does evidence exists that the results of same users completed once, and then again several weeks later, produce similar scores?14 weeks after the first TAGG completion a

large correlation was found with the second administration for all three version .80 for professional TAGG .70 for family TAGG .70 for student TAGG

Fairness Validity Evidence

Do differences exist by gender and racial groups?No overall differenceDifferences with employment construct

Do differences exist by SES?No overall difference by SES

Match Across Versions Validity Evidence

How closely do the different versions assess?TTAGG has medium correlations,

which is excellent for an assessment of this type

Concurrent Validity Studies

Match with TAGG and AIR Self-Determination AssessmentUnderway – data collected

Match with TAGG and Employer Identified Trait AssessmentUnderway – data collected

You Decide . . .Does the TAGG Fit Your Use?

Based on the given validity evidenceDoes the TAGG has ample evidence to

support how you want to use the results to build the PLAAFP and annual transition goals?

On-Line TAGG

The on-line TAGG will go live to the public before the end of this year

It will look something like this: http://zarrowcenter.ou.edu/taggp4/

TAGG Professional Version sample questions for the strengths and limitations construct. 1. The student told someone

what he or she does well.2. The student told someone

what he or she has trouble doing.

Sample Disability Awareness construct questions.1. The student expressed the

type of support or accommodations needed for his or her disability.

Sample items for the persistence construct.1. The student views not giving up

in school as important.2. The student keeps working to

achieve a goal, even when it becomes hard.

Sample items for the Interacting with Others construct.1. The student successfully participates in small groups to complete projects.

Sample items for the goal setting and attainment construct.1. The student creates short-term goals to attain long-term goals.2. The student adjusts plans to attain goals if they do not work.

Sample items for the employment construct.1. The student expresses wanting a job.2. The student had an unpaid job, such as

working for a family member.3. The had a paid job.

Sample items for the involvement in the IEP construct.1. The student told the IEP team

his or her postschool goals2. The student discussed his or

her present level of performance at the IEP meeting.

Sample items for the support community construct.1. The student accepts help

from support people when offered.

2. The student seeks assistance from community agencies.

Disability Awareness Profile

Sample disability awareness profile bar graph of results for the student, parent, and teacher TAGG versions. The graph is present on a 0 to 9 stanine profile.

Combined Score Profile

Sample profile bar graph of overall results for the student, parent, and teacher users. The graph is present on a 0 to 9 stanine profile.

Greatest and Relative Strengths

Summary of greatest strengths and areas of relative strengths, along with area of greatest need by student, family, and teacher user.

Areas of Greatest and Relative Need

Summary of greatest need, areas of relative need, and a summary of the overall TAGG results. Each can be copied and pasted into the IEP.

Summary Statement for IEP

Chad Bailey’s skills were assessed using the TAGG, a norm-referenced assessment with research-based items know to be associated with post-school employment and education. Compared to similar students Chad’s scores are average. Results indicate greatest strengths are in the areas of Goal Setting and Attainment. Chad’s relative strengths include Disability Awareness and Student Involvement in the IEP. Greatest needs are in the area of Strengths and Limitations, with Employment being a relative need.

Suggested Annual Transition Goals

To prepare for success in employment, the student will write an essay describing three situations where the student used his or her strengths with 90% accuracy in the areas of grammar and context by the end of the essay writing unit. Listing of Common Core Standards match to an annual goal. These are presented by grade for difference English benchmarks.

On-Going Studies

Continue additional validity studies, including SES and TAGG results

Follow group of students from high school into adult life

Determine relation between high school TAGG profile and adult outcomes

Establish criterion levels for constructsProduce profile with prioritized annual

transition goals based upon student’s profile and norm group outcomes

TAGG Availability

By the end of this fall semesterCan be found at the Zarrow Center web site

Google Zarrow CenterMinimal Cost

$3 per set (Professional, Student, Parent versions)

Minimum order 5 setsFunds used to keep TAGG on-line and

updated