CLEI

© 2008 K-CAT · All rights reserved2005 Research Park Circle · Manhattan, KS 66502www.k-cat.org

College Learning Effectiveness Inventory

Administration andScoring Manual for CLEI

ADMINISTRATION AND SCORING MANUAL FOR

THE COLLEGE LEARNING EFFECTIVENESS INVENTORY (CLEI)

by

Fred B. Newton, Eunhee Kim, Dan Wilcox, and Nathan Beemer

Kansas State University

Contributions in the development of the CLEI include recognition to

Ann Johnson, Wen-Chih Tseng, Kang-Hyun Shin, Mary Elizabeth Yeager, Ron Downey,

Stephen Benton, and participating staff from the KSU Counseling Services

2008 Edition

Copyright 2008 by Kansas State University Research Foundation

2

C O N T E N T S

Chapter 1 Overview ………………………………………………….………… 4

Introduction

Purpose

Item Development

Data Collection Procedures

Chapter 2 Administration and Scoring ……………………………………… 7

Administration

Appropriate Use

User Qualification

Scoring Procedures

Chapter 3 Interpretation of Results …………………………………………. 11

Interpretation of the Six Scales

Profile Interpretation

Chapter 4 Psychometric Information ………………………………………... 16

Sample Description

Exploratory Factor Analysis

Confirmatory Factor Analysis

Reliability

Validity

Chapter 5 Summary, Future Research, & Limitations …………………….. 19

References

Tables

3

Tables

Table 1 Items of the Six-CLEI Scales

Table 2 Means and Standard Deviations for the CLEI Scores for a Normative

Sample of College Undergraduates by Gender

Table 3 Means and Standard Deviations for the CLEI Scores for a Normative

Sample of College Undergraduates by Year in College

Table 4 Item Statistics for the Academic Self-Efficacy Scale

Table 5 Item Statistics for the Organization and Attention to Study Scale

Table 6 Item Statistics for the Stress and Time Press Scale

Table 7 Item Statistics for the Involvement with College Activity Scale

Table 8 Item Statistics for the Emotional Satisfaction Scale

Table 9 Item Statistics for the Class Communication Scale

Table 10 Scale Statistics: Mean, Standard Deviations, Intercorrelations, and Reliability

Coefficients with 95% Confidence Intervals for the Six Scales of the

CLEI

Table 11 Overall Fit Indices of the Confirmatory Factor Analysis on the Six Scales of

the CLEI

Table 12 Relationship between the CLEI and Validation Instruments

4

Chapter 1: Overview

Introduction

Counselors, academic advisors, and other professional support personnel at academic

institutions have a long history of using assessment to determine students’ academic ability

and achievement potential. Common practice has been to use ability tests and past academic

performance as standard measures. However, while ability and achievement measures are

important factors in predicting a student’s academic success, these assessments do not

explain a significant amount of individual variability in performance. Research summaries

by Astin (1993), Pascarella and Terenzini (1991), Russell and Petrie (1992), and Tinto (1992)

support the premise that student learning is impacted by many forces that include academic,

personal, social, and environmental variables. Russell and Petrie (1992) use three category

labels to describe factors: academic, social/environmental, and personality. These labels

served as a guide during the initial development of items.

Several writers have described principles of learning, based upon individual behavior

and impact of the learning environment that affect performance outcomes. Angelo (1993)

discusses thirteen research-based principles that can guide efforts to maximize learning.

Others have shown how specific personal factors can be directly influenced through

intervention strategies (Halstead, 1993; Newton and Smith, 1996; Newton, 1990). These

variables associated with the person within the context of campus experience are described as

psychosocial factors. They include personal, social, and environmental variables such as

aptitudes and abilities, attitudes, motivation, study approach, vocational interests, utilization

of campus resources, and sources of personal support available and utilized.

Critical to the identification of these variables has been the determination of the

relationship between the psychosocial variables and outcome performance. Identifiable

outcomes have included GPA, persistence or attrition in academic enrollment, satisfaction

with college life, adjustment defined by emotional wellbeing, self and other judgment toward

achieving goal success, and positive change over time measuring specified criteria.

Empirical studies of influence factors have provided evidence to suggest that relationships

exist between such variables and college performance (Chemers, Hu, & Garcia, 2001;

Davidson & Beck, 2006; Friedlander, Reid, Shupak & Cribbie, 2007; Lahmers & Zulauf,

2000; Macan, Shanhani, Dipboye & Phillips, 1990). Understanding the relationship between

certain psychosocial variables and success outcomes provides the opportunity for designing

educational and supportive strategies. Because of this relationship between psychosocial

influences and academic success, we believe that academic services personnel, with some

relevant training in both assessment and intervention, can make a significant difference in

student success in both the classroom and the overall campus environment.

Related to psychosocial influence on academic performance tasks, researchers have

evaluated the impact of specific variables such as time utilization (Lahmers & Zulauf, 2000;

Macan, Shahani, Dipboye & Phillips, 1990; Nonis & Hudson, 2006), strategic organization

and study approach (VanZile-Tamsen, 2001), academic self-esteem, efficacy, and confidence

(Chemers, Hu & Garcia, 2001; Friedlander, Reid, Shupak, & Cribbie, 2007; Lent, Brown, &

Larkin, 1984; Zajacova, Lynch, & Espenshade, 2005), stress and emotional components

(Davidson & Beck, 2006; Pritchard & Wilson, 2003), student involvement with campus life

(Anaya, 1996; Cooper, Healy, & Simpson, 1994), and motivation and task relevance (Bong,

5

2004; VanZile-Tamsen, 2001). Background research on these types of variables provides a

basis for identifying the need for an assessment instrument that can provide a quantitative

measure of the attitudes, behaviors, and dispositions that lead to success outcomes.

Purpose

The College Learning Effectiveness Inventory (CLEI) was conceived as a method to

measure some of these psychosocial factors that impact a student’s learning. The CLEI is an

inventory of six scales with 50 questions representing a continuum of individual behaviors,

attitudes, and dispositions related to academic activity. The purpose of the inventory is to

organize the self-reported student responses into thematic domains or categories that have

shown to contribute to academic success. These domains comprise the six scales of

measurement on the CLEI. After students respond to the items they immediately receive a

profile that reports their overall pattern including strengths and weaknesses. The initial

profile provides some immediate explanation of results to the student, however, it has greater

utility when used as an advising and planning tool by student service personnel working with

students to make improvements to optimize their learning approach. The major objectives of

creating the CLEI were to:

(a) Develop a series of clearly definable and operational items that a student could

use to measure their own thoughts, feelings, or behaviors related to academic

pursuits;

(b) Include a continuum, that reflects the degree from positive to negative, thereby

showing how the item content might support or interfere with academic

pursuits;

(c) Utilize an on-line survey format for ease of access, user-friendly administration

and rapid retrieval of results;

(d) Provide immediate feedback and information to the student completing the

inventory by showing a pattern of strengths and weakness on an individualized

profile;

(e) Provide information for advising and counseling a student, making it a tool for

discussion of goals, selection of interventions, referral to relevant student

services, and a measurement of progress and involvement in the change process;

(f) And, to utilize the CLEI as a tool for research describing relationships between

variables, measures of change, and outcome comparisons.

Item Development

The CLEI was developed over the past ten years in two stages. Initially, three

professional counseling staff, knowledgeable in the literature on learning, began generating

behavioral statements that would serve as potential measures of the six categories that were

seen as important to success. These categories included motivation, self-concept, study

habits, emotions, support, and involvement. Eventually, an item pool of over 300 items was

generated. After these items were identified a panel of nine expert judges, with experience

and credentials related to student learning, went through a systematic process to refine and

reduce these items. The panel judged each question on three criteria: (1) clarity and relevance

to college students, (2) accurateness and goodness of fit to the operational definitions of the

6

categories, and (3) placement of the item on a five point Likert scale (a score of “1” equals a

high negative behavior and “5” represents a high positive behavior). The intent of the judging

process was to eliminate non-discriminating and invalid questions. This process resulted in

the identification of 144 items covering the six categories and ranging from positive to

negative in wording.

The first generation CLEI with 144 items was piloted with over 500 students and

utilized with student “academic assistance groups” over a period of five years. Results from

the initial piloting of the CLEI found it to be a useful tool for counseling and advising

students on their academic problems. However, a review of this first version identified the

following problems: repetitive items similar in nature and the large number of items took too

much time to complete the inventory. An exploratory factor analysis was used to identify

and eliminate redundant questions and questions that lacked predictive value. This resulted

in the second version of the CLEI with 61 items compiling the six scales (Newton, Kim,

Wilcox, & Yeager, 2007).

The second version of the CLEI was tested with over 1,000 students for additional

studies including confirmatory factor analyses, reliability studies, and validity studies. It was

also tested to examine a scoring system utilizing T-Scores based on normative scores. These

studies suggested that the scales could have stronger psychometric properties by respecifying

the scales. Examinations of normative score distributions and individual response patterns

on each of the scales indicated that there was a slight positive skewing on several scales but a

significant skewing in positive direction for the Academic Self-Efficacy Scale. This

suggested that utilizing T-Scores could provide better interpretation of the individual profile

of the scales. Revisions were then made to the second version of the CLEI and yielded the

current 50 item, six scale version (2008 Edition).

The current College Learning Effectiveness Inventory (CLEI) consists of six scales.

This 50-item assessment tool measures individual attitudes and behaviors that may impact

academic performance. The CLEI serves both diagnostic and prescriptive functions.

Academic advisors, counselors and others whose work involves supporting student success

and retention can use the CLEI to assess individual student’s strengths and weaknesses.

Interventions can then be custom designed to address each student’s weaknesses, build upon

their strengths, and enhance their chances of success.

In addition, demographic variables are included with this on-line version of the CLEI.

These variables are gender, age, year in school, overall GPA, academic major, ethnic

identity, and residence type. Each college or university using the CLEI may elect to identify

demographic variables that are of particular interest in their on their campus.

Data Collection Procedures

Data for developing this revised CLEI and for developing normative profiles were

collected from undergraduate students who were enrolled in a large public university

(enrollment greater than 20,000 students) in the Midwest. Participants completed the online

CLEI via the Internet at remote locations during academic years 2006 and 2007. Sample

descriptions are presented in Chapter 4. These studies were approved by the Committee for

Research Involving Human Subjects under the University Research Compliance Office at

Kansas State University.

7

Chapter 2: Administration and Scoring

Administration

The CLEI is administered on-line via the Internet and scored automatically. Students

are given a password in order to access and take the CLEI. The inventory can usually be

completed in 10 to 15 minutes. Individual results are provided immediately after the CLEI is

completed. Students can print out their results and are provided a brief interpretative

summary. It is recommended that students also work with an advisor, counselor, trained peer

mentor, or faculty and staff familiar with the CLEI in order to utilize the results most

effectively. A follow-up interview or group explanation session is a way to review student

CLEI profiles, discuss the results, and begin to set goals and make recommendations for

action. The process follows the basic problem solving model utilizing the steps of “what”,

“so what”, and “now what”. Alternatively stated, the goal for students using the CLEI for

academic enhancement is to utilize the process of identification, implication, and

improvement.

Appropriate Use

The CLEI can be used in the following ways:

(1) As an assessment tool that helps each student become aware of attitudes and

behaviors that affect their learning and studying.

(2) As an organizing assessment that identifies specific areas in which each student

could benefit most from interventions.

(3) To develop specific interventions designed to remediate weaknesses and

capitalize on strengths.

(4) In pre-post measure to determine the effectiveness of specific interventions, and

to determine if additional interventions are needed.

(5) As a counseling or psychoeducational strategy for college orientation, advising,

educational development, and learning skills programs.

User Qualifications

The CLEI is designed as an easy to use tool that organizes the pattern and potential

meaning of a student’s self report on personal behaviors, attitudes, and feelings toward

academic activity. At the most basic level it can become an information source for the

student to view their own personal motivations and approach to academic activity.

Definitions, examples, and suggestions for follow-up options are provided on the profile

screen. The questions asked are not psychologically sensitive and the terminology does not

reflect concern for pathology or danger. The CLEI may also be used as a source of input for

advising or counseling a student on their approach to learning. It is recommended that the

best results in the use of CLEI will come from a professional or paraprofessional staff person

who has training with the assessment of academic concerns and is familiar with the

development and theory behind the CLEI.

8

An important supplement to CLEI results is the Student Learning Effectiveness

Workbook. The workbook outlines a step by step process to translate CLEI profile results

into personal action plan to make individualized improvements in one’s approach toward

learning.

9

Scoring Procedures

All scoring is generated automatically on-line by the use of formula based upon the

scale construction explained in Chapter 4. Immediately after the CLEI is completed,

individual results are provided to the student in the form of a profile chart. An individual

profile displays individual’s mean raw scores and T-Scores for each of the six scales. Prior

to calculating individual scores, items in negative continuum are transformed to reverse

scores (i.e., a score of 1 is transformed to a score of 5, score 2 to 4, score 4 to 2, score 5 to 1).

Table 1 provides a list of items for each of the six scales, and items with asterisk (*) indicates

items in negative continuum. Items are grouped and ordered by the loading size on each

scale to facilitate interpretation. The mentor working with a student can administratively

obtain an individual score sheet of the student that contains raw item scores, mean scores,

and T-Scores in addition to the profile chart.

Raw Mean Scores. The purpose of providing an individual raw mean score for each

of the six scales is to provide an interpretation that demonstrates an individual’s profile of

high and low scores indicating strengths and weaknesses from an intrapersonal perspective.

The mean score for each scale can vary from 1.0 (lowest possible score) to 5.0 (highest

possible). A low score reflects a negative response to the attributes of the scale, and a high

score represents a positive response to the attributes of the scale. A raw mean score will fall

somewhere between these two extremes. A score of 3.0 is the mid-point and any score

within the middle range of between 2.6 and 3.4 is more neutral and less likely to reflect a

strength or weakness. However, it should be noted that on any scale a mean score of 3.5 or

above would indicate a more positive response pattern and considered an area of personal

strength. A scale score of 2.5 or below would reflect more negative response or potential

area of weakness.

T-Scores. An individual student’s mean raw score for each scale is compared with

average scale scores generated by a normative sample, and T-Scores are calculated for each

of the six scales. Means and standard deviations (SD) of scores from the normative sample

are used to transform an individual student’s mean scores into T-Sores for each of the scales

using the following formula:

T-Score = 10*[(Individual Mean – Normative Mean)/SD of Normative Score]+50.

T-Scores are interpreted using a mean of 50 and a standard deviation of 10:

• 68% of the normative group falls between T=40 and T=60

• 96% of the normative group falls between T=30 and T=70

• 99% of the normative group falls between T=20 and T=80.

Typically, the T-scores for each scale will distribute in a manner called the normal inverted U

curve. This means more students cluster around the middle or average score of 50 with

fewer students as the scores move away from the middle. The one exception to this rule

when interpreting the CLEI scales is the first scale called Academic Self-Efficacy. The

10

majority of students rate their responses positively on this scale. As a result, a score around

50 to slightly below may still reflect a high positive self rating.

Normative Scores. The means and standard deviations for the CLEI scales obtained

from a normative sample are reported in Table 2 and Table. The participants in the

normative sample were university students (N=879) who were enrolled in a large public

university in the Midwest during academic years of 2006-2007. The normative scores for

each of the six scales are presented by the demographical factors of gender (Table 2) and

year-in-college (Table 3). This normative sample was weighted to reflect proportions of

male and female students. It was also weighted to reflect class level proportions in the

university population. The weighted sample consisted of 439 male students (49.9%) and 440

female students (50.1%). With regard to class level, 24.0% were freshmen (N=211), 21.1%

were sophomore students (N=186), 22.0% were juniors (N=193), and 33.0% were seniors

(N=290).

11

Chapter 3: Interpretation of Results

Interpretation of the Six Scales

! Academic Self-Efficacy (ASE Scale): Items on this scale measures an expression of

confidence in academic ability, awareness of effort toward study, and expectations of

success in college attainment. Those who score high have expectations to succeed

and accomplish important outcome goals. Those who score low are more likely to

feel uncertain about possible achievement and what the future may hold.

*Note: The Academic Self-Efficacy scale is skewed toward overall positive

responses, therefore, caution should be made in interpretation recognizing that a T-score

of 40 would be below the 50% of the norm group but would still reflecting a

positive overall response on the scale items.

! Organization and Attention to Study (OAS Scale): This scale measures the

organization of tasks and structuring of time to set goals, plan, and carry out

necessary academic activity. Those who score high are likely to use effective

organizational planning and time management skills to achieve academic success.

Those who score low are more likely to avoid planning strategies and lack focus of

attention in providing self-direction.

! Stress and Time Press (STP Scale): This scale measures how student deals with the

pressures of time, environmental concerns, and the academic demand that impacts

academic study. Those who score high manage the pressures of academics without

reactions such as being overwhelmed, procrastination, or avoidance. Those who

score low may experience symptoms of stress and do not believe they can handle the

academic demands they experience.

! Involvement with College Activity (ICA Scale): Involvement is defined by the ICA

Scale as belonging to organizations and participating in activities. Those who score

high belong to many organizations and often participate in formal and/or informal

campus activities. Those who score low are more socially isolated and are less likely

to participate or engage in campus activities.

! Emotional Satisfaction (ES Scale): This scale measures the degree of interest and

emotional response in academic life including people and the campus educational

environment. Those who score high express encouragement, interest, and positive

anticipation for academic life, whereas those who score low are more likely to

express discouragement, negative reactions, and a sense of being overwhelmed with

academic life.

! Class Communication (CC Scale): Communication in the CC Scale includes both

verbal and non-verbal effort to engage in class activity. Those who score high scorers

are assertive and active with written and oral communication in-class and with their

12

instructors. Those who score low may experience uncertainty and reluctance in

expressing and asserting their ideas in-class and with their instructors.

13

Profile Interpretation

Profiles from the CLEI may be best used as stimulus for discussion with the student.

One approach is to examine the individualized summary noting the position of scale scores

from high to low. To provide examples of profile interpretation, Figure 1, 2, and 3 illustrate

profile charts for three students.

Example Profile A. The profile chart A is for a female student, who is a sophomore,

majoring in Business/Marketing with a GPA of 3.0-3.4 on a 4.0 scale. Her two highest

scores are in Academic Self-Efficacy and Involvement with College Activity with a mid-

range score in Class Communication. Her lower scores are in Emotional Satisfaction,

Organization and Attention to Study, and Stress and Time Press. These scores indicate that

she is a confident and out-going student who is likely to be very active on campus and

engaged in her college experience. However, these results also suggest that her strengths

may create problems such as being pressed for time, feeling pressure to become better

organized and having some worry or stress about academics.

Fig. 1. Example Profile Chart A

Profile A

14

Example Profile B. The profile chart B is for a female student, who is a junior,

majoring in Elementary Education with a GPA of 2.5-2.9 on a 4.0 scale. The student’s

profile is marked by low scores on the Scales 2, 3, and 4 (lack of organization and attention

to study, stress and time press, and low involvement with college activity). The student is in

the mid-range on Academic Self-Efficacy and Emotional Satisfaction Scales. The reason for

stress and time press, lack of organization and attention to study, and low activity level may

reflect the presence of some situational stressor occurring at this time in her life. In some

cases, it might be family issues, the need to work extra hours to meet financial obligations, or

a personal problem that is interfering with college life. Is this a situation stressor occurring

in her life? Or, does she have a history of anxiety or stress when preparing her assignments

or when performing tasks? Exploring these contrasting scores with the student could help to

identify possible solutions or result in referral to the appropriate service or support program

on campus.

Fig. 2. Example Profile Chart B

Profile B

15

Example Profile C. This sophomore male student is struggling in college with a

grade point average below 1.9. The profile shown in Figure 3 indicates flat and below

average scores across five of the scales (1, 2, 4, 5, & 6). Scale 3 indicates a fairly average

amount of stress or time pressure on the individual. This suggests several possibilities to

explore. One area might be competing involvement in work, family, or other activity

unrelated to college that distracts the student from sufficient engagement to succeed in

academic work. Another possibility could reflect uncertainty as to whether college is desired

or necessary to achieve his personal goals. Career planning, personal decision-making, or

more exploration could be important for this student to find a successful niche.

Fig. 3. Example Profile Chart C

Profile C

16

Chapter 4: Psychometric Information

Sample Description

Samples of university students were used in three separate studies: (a) a derivation

sample for an exploratory factor analysis (EFA), (b) a replication sample for a confirmatory

study (CFA), and (c) a validation sample for a cross-validation study. Participants were

university undergraduate students who were enrolled in a large public university (enrollment

greater than 20,000 students) in the Midwest.

Derivation Sample. A sample of 587 university undergraduate students was used in

the derivation sample for EFA. The average age of the participants was 21.21 (SD= 4.33),

with an age range of 17 to 56 years. About two-thirds of the participants were women

(N=405, 67.8%). Class levels included freshmen (51.4%), sophomores (12.4%), juniors

(16.4%), and seniors (19.8%). Reported overall Grade Point Averages (GPA) were: 3.5 and

above (33.9%), 3.0 to 3.4 (32.9%), 2.5 to 2.9 (21.7%), 2.0 to 2.4 (6.5%), and below 2.0

(5.0%). Ethnically, 86.6% reported that they were Caucasian, 3.4% were Hispanic/Latino

American, 3.0% were African American/Black, 1.5% were Asian American, 0.3% were

Native American, and 0.2% were multicultural. Majority students lived off campus (55.3%)

and 44.7% lived on-campus.

Replication Sample. A sample of 282 university undergraduate students was used in

the replication sample for CFA. The average age of the participants was 20.91 (SD= 3.71),

with an age range of 18 to 45 years. In this sample, 73.4% were women (N=207). Class

levels included freshmen (36.9%), sophomores (4.6%), juniors (20.6%), and seniors (37.9%).

Reported overall Grade Point Averages (GPA) were: 3.5 and above (30.9%), 3.0 to 3.4

(31.2%), 2.5 to 2.9 (18.8%), 2.0 to 2.4 (3.5%), and below 2.0 (0.4%). Regarding ethnicity,

90.4% reported that they were Caucasian, 1.8% were Hispanic/Latino American, 2.1% were

African American/Black, 1.4% were Mexican American, 0.7% were Asian American, 0.7%

were Native American, and 2.8% were multicultural. Majority students lived off campus

(60.3%), 29.1% lived on-campus, and 10.6% lived at fraternity/sorority houses.

Validation Sample. A sample of 160 university undergraduate students was used in

the construct validity study. The average age of the participants was 22.44 (SD= 4.74), with

an age range of 19 to 49 years. In this sample, 73.1% were women (N=117). Majority of the

participants were seniors (61.9%), 33.1% were juniors, and 3.8% were sophomores. There

were no freshmen among the respondents. Most students participating in the validation study

have a cumulative grade point average of greater than or equal to 3.00 (77.9%). Reported

overall Grade Point Averages (GPA) were: 3.5 and above (39.9%), 3.0 to 3.4 (38.0%), and

2.5 to 2.9 (20.3%). None of the participants had their GPA below 2.5. Ethnically, most of

the participants were Caucasian (93.1%), and minority group consisted of Hispanic/Latino

American or Mexican/Mexican Americans (2.4%), African Americans (1.2%), Asian/Pacific

Islander (1.2%), Native American/Alaskan Native (1.2%), and multiracial (0.6%). Majority

students lived off campus (83.8%), 4.4% lived on-campus, and 8.8% lived at

fraternity/sorority houses.

17

Exploratory Factor Analysis

An exploratory factor analysis was conducted on a derivation sample (N=597) to

identify underlying factors of the 62 items of the CLEI. We performed a principal

component extraction with promax (oblique) rotation of the correlation matrix from the

derivation data. Based on the factor analysis and conceptual analysis by the expert panel, we

adopted the revised CLEI version comprised of six scales and 50 items. Summaries of the

item statistics can be found in Table 4 through Table 9 for each of the six scales. Items are

ordered and grouped by size of coefficients to facilitate interpretation. Table 10 presents

scale statistics including means, standard deviations, intercorrelations, and reliability

coefficients with 95% confidence intervals for the six scales of the CLEI. The scale labels

with number of items and reliability coefficients of internal consistency for these six scales

are provided below.

(1) Academic Self-Efficacy (ASE Scale: 14 items, Cronbach’s Alpha=.87)

(2) Organization and Attention to Study (OAS Scale: 8 items, Cronbach’s Alpha=.81)

(3) Stress and Time Press (STP Scale: 6 items: Cronbach’s Alpha=.77)

(4) Involvement with College Activity (ICA Scale: 9 items, Cronbach’s Alpha=.81)

(5) Emotional Satisfaction (ES Scale: 7 items, Cronbach’s Alpha =.72)

(6) Class Communication (CC Scale: 6 items, Cronbach’s Alpha=.68)

Confirmatory Factor Analysis

To examine whether the six factors of the CLEI from the derivation sample of the

exploratory factor analysis adequately applied to the replication sample, we conducted a

confirmatory factor analysis on a replication sample (CFA sample, N=292) using unweighted

least squares estimate and correlated errors within scales estimation. Seven fit indices were

used to determine the data fit of the hypothesized model: (a) chi-square (Satorra-Bentler

Scaled Chi-Square), (b) normed fit index (NFI), (3) goodness-of-fit index (GFI), (d) adjusted

goodness-of-fit index (AGFI), (e) root mean square residual (RMSR), (f) root mean square

error of approximation residual (RMSEA), and (g) comparative fit index (CFI). As shown in

Table 11, overall fit indices indicated that the CFA for the six CLEI scales resulted in a

generally acceptable fit.

Reliability

The internal consistency of scores for the six scales was examined on the EFA sample

(N=597) and CFA sample (N=292). The respective Cronbah alphas for the EFA and CFA

samples were: ASE (.87 and .86), OAS (.81 and .81), STP (.77 and .71), ICA (.81 and .71),

ES (.72 and .74), and CC (.68 and .68). Trends in reliability alphas for the exploratory

sample and for the confirmatory sample were consistent. Scores for five of the six scales had

reliabilities that were adequate ranging from .71 to .87, while scores for the Class

Communication Scale had marginal reliabilities for both EFA and CFA samples. Reliability

coefficients with 95% confidence intervals for the six scales were presented in Table 10.

18

Validity

To examine the degree to which the six scales measure what they were intended to

measure, construct validity was tested on a validation sample (N=160). The validation

instruments used were: (a) selected subscales of the Learning and Study Strategies Inventory

(LASSI) (Weinstein & Palmer, 2002), (b) Rosenberg Self-Esteem Scale (Rosenberg, 1965),

(c) Crombag College Adaptation Questionnaire (Van Rooijen, 1986), and (d) Student

Propensity to Ask Questions (Cunconnan, 1996; Cayanus, 2005).

Correlations between the validation instruments and the CLEI scales are shown in

Table 12. The Academic Self-Efficacy Scale of the CLEI was correlated with both the

Motivation Scale of the LASSI (r=.46) and Rosenberg Self-Esteem Scale (r=.45). The

Organization and Attention to Study Scale showed correlations with two subscales of the

LASSI: Concentration Scale (r=.71) and Self-Testing Scale (r=.46). The Stress and Time

Press Scale was correlated with the Time Management Scale of the LASSI (r=.44). The

correlation coefficient between the Involvement with College Activity Scale and the Student

Adaptation to College Questionnaire was a bit low (r=.31), but it was statistically significant.

The Emotional Satisfaction Scale was correlated with the Attitude Scale of the LASSI

(r=.50), and the Class Communication Scale was correlated with the Student Propensity to

Ask Questions (r=.53).

19

Chapter 5: Summary, Future Research, & Limitations

Summary

The CLEI is an instrument designed to measure individual attitudes and behaviors

that may impact academic performance. This instrument was developed because a group of

professionals (counselors and advisors) identified the need for a tool to explore what was

described as personal/social influences that impacted learning performance.

Over 300 items were originally developed through both the inductive experience in

working with students and through the collective published information on college student

learning. During the past several years the instrument has gone through a process to clarify

and define the psychometric properties of the instrument. The present edition of the CLEI

(2008 Edition) contains 50 items that load on to six scales measuring positive and negative

levels of student attitudes and behaviors.

The CLEI can function as a tool of exploration for students to increase their

awareness of personal strengths and weaknesses as a learner and be able to make changes

that may improve academic performance. It may be most effective with professionals trained

to assist students with academic problems (counselors, advisors, learning center staff).

However, it is also available as a self-help on-line inventory that provides individual students

feedback and suggestion on their learning effectiveness.

Future Research

The CLEI should be considered as an instrument in development. There are several

areas of research that are needed to contribute to the understanding and use of this

instrument. The followings are suggestions for research, studies that are either in process or

anticipated within the near future. Results from these studies will be reported at a later date

and included within this manual.

1. A larger and more diverse sample of students to be tested to establish a

normative profile of information. This sample would be drawn from

institutions representing different geographical areas of the country, different

institutional types (private, public, 2-year), size, and degree emphasis.

2. Comparisons between identifiable sub-groups of students to discriminate

response patterns including demographic profile and academic definition

clusters (gender, major, year in school, ethnicity, first generation, probation

status, honor student status, etc.).

3. Additional studies of psychometric properties including test-retest

examination of the instrument’s stability, and comparison of a group of

students and a group of advisors to examine convergent validity.

4. Utilizing a complete battery of the three K-CAT inventories to provide a

broad picture of college students’ attitudes, behaviors, and health patterns

related to their overall functioning in college. These three inventories (K-

20

PIRS, HBA, & CLEI) measure three dimensions of health behaviors, mental

health, learning attitudes and behaviors providing a comprehensive view of

psycho-social variables that may impact student performance. The suggestion

is to utilize testing of freshmen students early in their college career and track

changes with academic outcomes through their college experience.

5. Utilizing the CLEI as a tool for prescribed programs of intervention that can

measure students at pre, post, and follow-up periods of time to determine the

efficacy of the instrument for tracking change.

Limitations

The CLEI is a tool to be used to identify potential strengths and weaknesses of

student attitudes and behaviors related to academic performance. As such, it should be used

for exploration, clarification, and a way to initiate conversation on ways to maintain or

improve their learning activity. The CLEI is not proven as an evaluation, diagnostic, or

prescriptive instrument. It should not be used to determine learning disabilities, eligibility

for accommodations, or as a basis for determination of success or failure decisions.

The CLEI has not been tested on many specific sub-populations of students and may

not generalize to the behavior and attitudes considered important for the success of students

in special or unique situations. For example, graduate students or distance learning students

may not be appropriate for many items that measure involvement in on-campus activities or

in-class performance activities.

21

REFERENCES

Angelo, T. A. (1993). A teacher’s dozen: Fourteen general findings from research that can

reform classroom teaching and assessment and improve learning. American Association of

Higher Education Bulletin, 3-8.

Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco:

Jossey-Bass, Inc.

Bong, M. (2004). Academic motivation in self-efficacy, task value, achievement goal

orientations, and attributional beliefs. The Journal of Educational Research, 97 (6), 287-297.

Cayanus, J. L. (2005). Student’s propensity to ask questions: Do cognitive flexibility, teacher

self-disclosure, student motives to communicate, and affective learning influence question

asking in the classroom? Unpublished doctoral dissertation, West Virginia University,

Morgantown.

Chemers, M.M., Hu, L. & Garcia, B. F. (2001). Academic self-efficacy and first-year college

student performance and adjustment. Journal of Educational Psychology, 93 (1), 55-64.

Cunconan, T. M. (1996). The conceptualization, measurement, and validation of a student’s

propensity to ask questions in the college classroom. Unpublished doctoral dissertation,

University of Oklahoma, Norman.

Davidson, W. B. & Beck, H. P. (2006). Using the survey of academic orientations to predict

undergraduates’ stress levels. NACADA Journal, 26 (2), 13-20.

Friedlander, L.J., Reid, G.j., Shupak, N. & Cribbie, R. (2007). Social support, self-esteem,

and stress as predictors of adjustment to university among first-year undergraduates. Journal

of College Student Development, 48 (3), 259-274.

Gibson, D.M. & Myers, J.E. (2006). Perceived stress, wellness, and mattering: A profile of

First-year Citadel cadets. Journal of College Student Development, 47 (6), 647-660.

Halstead, R. W. (1993). The 16 principles: A guide to getting better grades in college.

Worcester, MA: COB Press.

Lahmers, A. G. & Zulauf, C.R. (2000). Factors associated with academic time use and

academic performance of college students: A recursive approach. Journal of College Student

Development, 41 (5), 544-556.

Lent, R.W., Brown, S.D., & Larkin, KC. (1984). Relation of self-efficacy expectations to

academic achievement and persistence. Journal of Counseling Psychology, 31, 352-362.

22

Macan, T. H., Shahani, C., Dipboye, R. L. & Phillips, A. P. (1990). College students’ time

management: Correlations with academic performance and stress. Journal of Educational

Psychology, 82 (4), 760-767.

Newton, F. B. (1990). Academic support seminars: A program to assist students

experiencing academic difficulty. Journal of College Student Development, 31(2), 183-186.

Newton, F.B., Kim, E., Wilcox, D. & Yeager, M.E. (2007). Administration and scoring

manual for the College Learning Effectiveness Inventory 2007 edition. Kansas State

University Research Foundation.

Newton, F. B. & Smith, J. H. (1996). Principles and strategies for enhancing student learning.

In. Ender, S. C., Newton, F. B. & Caple, R. B. Contributing to learning: The role of student

affairs. San Francisco: Jossey-Bass, Inc.

Nonis, S. A. & Hudson, G. I. (2006). Academic performance of college students: Influence

of time spent studying and working. Journal of Education for Business, 81 (3), 151-159.

Pascarella, E. T. & Terenzini, P. T. (1991). How college affects students: Findings and

insights from twenty years of research. San Francisco: Jossey-Bass, Inc.

Pritchard, M.E. & Wilson, G.S. (2003). Using emotional and social factors to predict student

success. Journal of College Student Development, 44 (1), 18-28.

Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton

University Press.

Russell, R. K. & Petrie, T. P. (1992). Academic adjustment of college students: Assessment

and Counseling. In. S.D. Brown and R.W. Lent (Eds.) Handbook of Counseling Psychology,

New York: Wiley.

Tinto, V. (1992). Leaving college: Rethinking the causes and cures of student attrition 2nd

edition. Chicago: University of Chicago Press.

Van Rooijen, L. (1986). Advanced students' adaptation to college. Higher Education.

15(3/4), 197-209. New York: Springer.

VanZile-Tmsen, C. (2001). The predictive power of expectancy of success and task value for

college students’ self-regulated strategy use. Journal of College Student Development, 42 (3),

233-241.

Weinstein, C.E. & Palmer, D.R. (2002). User’s manual: Learning and study strategies

inventory 2nd

edition, H&H Publishing Company, Inc.

Zajacova, A., Lynch, S. M. & Espenshade, T. J. (2005). Self-efficacy, stress and academic

success in college. Research in Higher Education, 46 (6), 677-706.

23

Table 1

Items of the Six-CLEI Scales

1. Academic Self-Efficacy (ASE Scale) (14 items) 23. I believe that I have the ability to complete college. 26. I have goals that I want to achieve by being in college. 43. I have high academic expectations of myself. 24. I believe it is possible for me to make good grades. * 28. I turn in assignments only partially completed. * 42. I doubt that I can make the effort to finish college. 50. I am determined to do what it will take in order to succeed with my goals. * 5. I do not turn in assignments. 21. My family cares how I do academically. * 38. Family members criticize me because I am not a great student. 4. I am aware of the assignments that are due in the next week. 47. Gaining knowledge is important to me. * 49. I question why I need a degree for the career I want to pursue. 20. People in my community value a college education.

2. Organization and Attention to Study (OAS Scale) (8 items) 2. I organize my time so that I have plenty of time to study. 30. I make study goals and keep up with them. * 1. I wait to study until the night before the exam. 31. I break big assignments into manageable pieces. * 51. I cannot get into studying even if there is nothing else to do. * 48. I find myself daydreaming when I study. * 25. I find my attention wandering in class 6. I organize class information in a way that helps me retain and apply it later.

3. Stress and Time Press (STP Scale) (6 Items) * 36. I feel there are so many things to get done each week that I am stressed. * 13. I have symptoms of stress from all of the pressure I have been under since coming to college. * 3. I do not seem to have time to get everything done that I need to do. * 32. It seems as though I am playing catch-up. * 37. My living situation distracts me from my studies. 7. I plan in advance to prevent becoming overwhelmed with assignments at the last minute.

24

4. Involvement with College Activity (ICA Scale) (9 Items) 9. I participate in social activities on campus. 11. I belong to an organized club on campus. 34. I attend events such as concerts, plays, speakers, or athletic contests as a part of the college experience. 29. I know someone with whom I can study. 40. I have friends here at school. 10. I belong to a study group. 15. I consider college to be a great time in my life. 41. My friends have good study habits. 17. I enjoy being a student here.

5. Emotional Satisfaction (ES Scale) (7 items) 14. I like my courses. 39. My instructors show interest in me. * 18. I hate school, but I know I have to do it. 27. I see connections between my classes and my career goals. * 12. I am discouraged with how I am treated by my instructors. 19. I can talk with people who provide encouragement to me about what I am learning. * 16. I become overwhelmed when I think of my assigned class requirements.

6. Class Communication (CC Scale) (6 Items) * 8. I avoid speaking in class. 33. I ask questions in class. * 46. I cannot seem to express my ideas on paper very well. * 35. I avoid classes in which participation is required. * 44. I dread the thought of getting test results in certain classes. * 22. I find it difficult to get the assistance I need for my academic success.

“Experimental Item” 45. I can make connections between what I learn in class and my plans for a career. Note. *, Indicates items in negative continuum of which the raw scores need to be transformed to reverse scores: 1 to 5, 2 to 4, 4 to 2, and 5 to 1.

25

Table 2

Means and Standard Deviations for the CLEI Scores for a Normative Sample of College Undergraduates by Gender

Gender CLEI Scales

Male Female Total

1. Academic Self-Efficacy Mean 4.25 4.54 4.40

SD .51 .43 .50

2. Organization and Attention to Study Mean 3.03 3.18 3.10

SD .60 .57 .59

3. Stress and Time Press Mean 3.16 2.98 3.07

SD .68 .67 .68

4. Involvement with College Activity Mean 3.41 3.51 3.46

SD .67 .60 .64

5. Emotional Satisfaction Mean 3.55 3.73 3.64

SD .55 .55 .56

6. Class Communication Mean 3.41 3.44 3.42

SD .64 .59 .62

** Global Score Mean 3.47 3.57 3.52

SD .42 .39 .41

Note. N=879 (Male n=439, Female n=440). Means and standard deviations are weighted by gender ratios (.512 for male student sample and .488 for female student sample) and by class level proportions in the larger university population (.242 for First Year, .205 for Sophomore, .225 for Junior, and .327 for Senior) in the larger university population.

26

Table 3

Means and Standard Deviations for the CLEI Scores for a Normative Sample of College Undergraduates by Year in College

Year in School CLEI Scales

First

Year

Sophomore

Junior Senior Total

Mean 4.26 4.34 4.54 4.44 4.40 1. Academic Self-Efficacy

SD .53 .55 .39 .47 .50

Mean 3.08 3.09 3.13 3.10 3.10 2. Organization and Attention to Study SD .60 .58 .63 .56 .59

Mean 3.17 3.15 2.96 3.02 3.07 3. Stress and Time Press SD .71 .64 .69 .68 .68

Mean 3.32 3.61 3.55 3.40 3.46 4. Involvement with College Activity SD .70 .66 .55 .60 .64

Mean 3.54 3.56 3.77 3.69 3.64 5. Emotional Satisfaction SD .57 .62 .49 .52 .56

Mean 3.27 3.38 3.49 3.52 3.42 6. Class Communication

SD .62 .68 .55 .59 .62

Mean 3.44 3.52 3.58 3.53 3.52 ** Global Score

SD .44 .44 .35 .39 .41

Note. N=879 (First Year n=211, Sophomore n=186, Junior n=193, Senior n=290). Means and standard deviations are weighted by gender ratios (.512 for male student sample and .488 for female student sample) and by class level proportions

(.242 for First Year, .205 for Sophomore, .225 for Junior, and .327 for Senior) in the larger university population.

27

Table 4

Item Statistics for the Academic Self-Efficacy Scale

Item

Number Item

Mean Item

Std. Deviation Item-Total

Correlation Cronbach’s Alpha

if Item Deleted

23 4.64 .697 .690 .855

26 4.55 .707 .623 .858

43 4.22 .850 .660 .855

24 4.42 .808 .631 .857

28 4.37 .797 .499 .864

42 4.65 .745 .595 .859

50 4.30 .801 .626 .857

5 4.45 .833 .483 .864

21 4.66 .688 .424 .867

38 4.32 1.006 .505 .864

4 4.07 .965 .392 .871

47 4.34 .790 .522 .862

49 4.20 1.074 .479 .867

20 4.39 .811 .426 .867

Table 5

Item Statistics for the Organization and Attention to Study Scale

Item

Number Item

Mean Item

Std. Deviation Item-Total

Correlation Cronbach’s Alpha

if Item Deleted

2 3.21 .888 .593 .777

30 3.07 .965 .572 .779

1 2.77 .946 .481 .793

31 3.19 .936 .512 .788

51 3.18 1.015 .610 .773

48 2.82 .897 .546 .783

25 2.73 .879 .482 .793

6 3.59 .864 .378 .806

28

Table 6

Item Statistics for the Stress and Time Press Scale

Item

Number Item

Mean Item

Std. Deviation Item-Total

Correlation Cronbach’s Alpha

if Item Deleted

36 2.76 1.059 .660 .694

13 3.04 1.102 .599 .710

3 3.12 1.060 .546 .725

32 3.01 1.023 .545 .726

37 3.19 1.194 .408 .764

7 3.14 .943 .329 .775

Table 7

Item Statistics for the Involvement with College Activity Scale

Item

Number Item

Mean Item

Std. Deviation Item-Total

Correlation Cronbach’s Alpha

if Item Deleted

9 3.28 1.122 .645 .768

11 3.03 1.627 .559 .786

34 2.74 1.152 .529 .783

29 3.49 1.123 .505 .787

40 4.45 .888 .542 .784

10 1.98 1.109 .381 .802

15 4.03 .925 .514 .787

41 3.54 .905 .395 .799

17 4.12 .926 .516 .787

29

Table 8

Item Statistics for the Emotional Satisfaction Scale

Item

Number Item

Mean Item

Std. Deviation Item-Total

Correlation Cronbach’s Alpha

if Item Deleted

14 3.52 .804 .531 .674

39 3.30 .899 .494 .679

18 3.57 1.142 .504 .675

27 3.72 .985 .444 .690

12 4.06 .900 .327 .717

19 3.73 1.017 .385 .706

16 3.41 .824 .392 .702

Table 9

Item Statistics for the Class Communication Scale

Item

Number Item

Mean Item

Std. Deviation Item-Total

Correlation Cronbach’s Alpha

if Item Deleted

8 3.07 1.089 .532 .592

33 2.90 1.027 .410 .637

46 3.58 1.047 .337 .662

35 4.07 .935 .416 .636

44 2.72 1.031 .350 .657

22 3.74 1.028 .410 .637

30

Table 10

Scale Statistics: Means, Standard Deviations, Intercorrelations, and

Reliability Coefficients with 95% Confidence Intervals for the Six Scales of the CLEI

N of Items

M SD Intercorrelation Coefficient Reliability Coefficient

Factor

1. ASE

2.OSA

3. STP

4.ICA 5.ES 6.CC

(95% CI)

1. ASE 14

4.40 0.51 --- .87

a (.855,

.885) 2. OSA

8 3.06 0.60 .45** ---

.81a (.784,

.831)

3. STP 6

3.04 0.73 .26** .41** --- .77

a (.738,

.796)

4. ICA 9

3.40 0.69 .43** .35** .18** --- .81

a (.782,

.829)

5. ES 7

3.62 0.58 .56** .51** .38** .38** --- .72 (.689, .757)

6. CC 6

3.34 0.64 .39** .36** .32** .35** .47**

--- .68 (.637, .717)

Note. N=597. ASE=Academic Self-Efficacy, OSA=Organization and Attention to Study, STP=Stress and Time Press, ICA=Involvement with College Activity, ES=Emotional Satisfaction, CC=Class Communication. a Cronbach’s alpha was significantly greater than hypothesized value of .70 (p<.05).

**p<.01.

Table 11

Overall Fit Indices of the Confirmatory Factor Analysis on the Six Scales of the CLEI

df NFI GFI AGFI RMSR RMSEA CFI

Independence Model Fit Statistics

22,447.46**

1.898.62**

1,225 1,118

.92

.92

.90

.08

.05

.96

Note. N=292. Fit indices include chi-square (Satorra-Bentler Scaled Chi-Square), NFI=normed fit index, GFI=goodness-of-fit index, AGFI= adjusted goodness-of-fit index, RMSR=root mean square residual , RMSEA=root mean square error of approximation residual, and CFI=comparative fit index. **p<.01.

Table 12

31

Relationship between the CLEI and Validation Instruments

CLEI Scales Validation Instruments Pearson r

1. Academic Self-Efficacy 2. Organization and Attention to Study 3. Stress and Time Press 4. Involvement with College Activity 5. Emotional Satisfaction 6. Class Communication

LASSI (Motivation Scale) Rosenberg Self-Esteem Scale LASSI (Concentration Scale) LASSI (Self-Testing Scale) LASSI (Time Management Scale) Crombag College Adaptation Questionnaire LASSI (Attitude Scale) Student Propensity to Ask Questions

.46** .45** .71** .46** .44** .31** .50** .53**

Note. N=160. LASSI=Learning and Study Strategies Inventory **p<.01.