florida state university libraries -...
TRANSCRIPT
Florida State University Libraries
Electronic Theses, Treatises and Dissertations The Graduate School
2014
The Psychometric Properties of the BarkleyDeficits in Executive Functioning Scale(BDEFS) in a College Student PopulationTheodora Passinos Coffman
Follow this and additional works at the FSU Digital Library. For more information, please contact [email protected]
FLORIDA STATE UNIVERSITY
COLLEGE OF EDUCATION
THE PSYCHOMETRIC PROPERTIES OF THE
BARKLEY DEFICITS IN EXECUTIVE FUNCTIONING SCALE (BDEFS) IN A COLLEGE
STUDENT POPULATION
By
THEODORA PASSINOS COFFMAN
A Dissertation submitted to the
Department of Educational Psychology and Learning Systems in partial fulfillment of the
requirements for the degree of Doctor of Philosophy
Degree Awarded:
Summer Semester, 2014
ii
Theodora Coffman defended this dissertation on May 6, 2014
The members of the supervisory committee were:
Frances Prevatt
Professor Directing Dissertation
Lee Stepina
University Representative
Beth Phillips
Committee Member
Debra Osborn
Committee Member
The Graduate School has verified and approved the above-named committee members, and
certifies that the dissertation has been approved in accordance with university requirements.
iii
ACKNOWLEDGEMENTS
I would like to thank Dr. Frances Prevatt for serving as my major professor, providing me
with support and encouragement during my graduate school career. I would also like to
recognize how smooth she made the process of doctoral training for me with her continued
guidance through every step of the way. I would like to thank my doctoral committee; Dr. Debra
Osborn, Dr. Beth Phillips, and Dr. Lee Stepina for their support throughout the dissertation
process and for being so giving of their time.
I thank my husband, Michael, for his continuous support over the years, and for his
willingness to listen to me talk about the ins and outs of executive functioning to exhaustion. I
thank my in-laws for their keen eye in catching typos in this document, and my parents for their
support. I would also like to thank all of my friends in the College of Education at FSU for their
continued support and willingness to order takeout at ALEC at all hours of the day and night to
make sure the work continued.
iv
TABLE OF CONTENTS
LIST OF TABLES ....................................................................................................................... vi
ABSTRACT ................................................................................................................................. vii
CHAPTER 1 .................................................................................................................................. 1
INTRODUCTION ........................................................................................................................ 1
Social Significance ...................................................................................................................... 1
Biological Factors Explaining the Relationship Between EF and ADHD .................................. 3
Barkley’s Model of EF and ADHD............................................................................................. 3
Assessing EF: Psychometric Tests or Impairment Rating Scales .............................................. 4
Test Construction ........................................................................................................................ 5
Statement of the Problem ............................................................................................................ 6
Research Questions ..................................................................................................................... 7
CHAPTER 2 .................................................................................................................................. 9
LITERATURE REVIEW ............................................................................................................ 9
ADHD and How the College Student is Different ...................................................................... 9
The Relationship Between ADHD and EF: How Biology and Neuropsychology Inform Our Understanding ........................................................................................................................... 15
EF Deficits as Manifested in ADHD......................................................................................... 25
Assessing EF ............................................................................................................................. 29
Test Construction and Validation Principles ............................................................................. 48
Proposed Study and Research Questions .................................................................................. 53
CHAPTER 3 ................................................................................................................................ 56
METHODS .................................................................................................................................. 56
Introduction ............................................................................................................................... 56
Hypotheses and Planned Data Analyses ................................................................................... 61
CHAPTER 4 ................................................................................................................................ 72
RESULTS .................................................................................................................................... 72
Demographic Variables and Statistics....................................................................................... 72
Research Question 1 .................................................................................................................. 72
Research Question 2 .................................................................................................................. 74
Research Question 3 .................................................................................................................. 75
Research Question 4 .................................................................................................................. 84
CHAPTER 5 ................................................................................................................................ 88
DISCUSSION .............................................................................................................................. 88
v
Relationship between BDEFS Self-Report Form and Other-Informant Form .......................... 88
Relationship between the BIA and BDEFS Factors.................................................................. 90
Reevaluation of ADHD-EF Index ............................................................................................. 93
Evidence of Factorial Validity .................................................................................................. 98
Limitations .............................................................................................................................. 100
Implications for Future Research ............................................................................................ 102
Implications for Clinical Practice ............................................................................................ 103
APPENDIX A ............................................................................................................................ 105
INFORMED CONSENT FOR ADHD GROUP .................................................................... 105
APPENDIX B ............................................................................................................................ 107
BARKLEY DEFICITS IN EXECUTIVE FUNCTIONING SCALES ................................ 107
APPENDIX C ............................................................................................................................ 111
INFORMED CONSENT FOR CONTROL GROUP ............................................................ 111
APPENDIX D ............................................................................................................................ 113
INTERNAL REVIEW BOARD FOR HUMAN SUBJECTS APPROVAL ........................ 113
APPENDIX E ............................................................................................................................ 115
DEMOGRAPHIC QUESTIONNAIRE .................................................................................. 115
APPENDIX F ............................................................................................................................ 116
BDEFS SCORING TEMPLATE ............................................................................................ 116
APPENDIX G ............................................................................................................................ 117
OTHER INFORMANT BDEFS .............................................................................................. 117
REFERENCES .......................................................................................................................... 121
BIOGRAPHICAL SKETCH ................................................................................................... 133
vi
LIST OF TABLES
1 Demographics ....................................................................................................................57 2 Suggested Ranges for Fit-Indices ......................................................................................71 3 Inter-Rater Correlations for College Student Sample, Comparing Self-Reports to Other-Reports .....................................................................................................................73 4 Inter-Rater Correlations for Barkley’s Sample, Comparing Self-Reports to Other-
Reports ...............................................................................................................................73 5 Means, t-Tests, and p-Value for Self vs. Other-Informant Forms, in the Current Sample................................................................................................................................74 6 Summary of Canonical Coefficients and Structure Loadings ............................................76 7 New 15-Item ADHD-EF Index ..........................................................................................80 8 Original 11-Item ADHD-EF Index ....................................................................................82 9 Summary of Canonical Discriminant Functions ................................................................82 10 ADHD-EF Index Description of Models ...........................................................................83 11 Functions at Group Centroids ............................................................................................83 12 Classification Rates ............................................................................................................83 13 CFA Models .......................................................................................................................85 14 Standardized Factor Loadings and Standardized Residual Variances ...............................85
vii
ABSTRACT
Approximately 4.4% of the adult population suffers from Attention-Deficit/Hyperactivity
Disorder (ADHD) (Keesler et al., 2010). The identification of adults with ADHD can be
difficult because the criteria in the Diagnostic and Statistical Manual of Mental Disorders (DSM;
APA, 2013) were originally designed with children in mind. Identifying high achieving college
students with ADHD has proven even more challenging due to masked academic difficulties
until later in life.
The specific population of adults in the college setting (college students) with ADHD are
more likely to have protective factors such as higher cognitive abilities and previous academic
success (DuPaul, Weyandt, O’Dell, & Varejao, 2009; Glutting, Youngstrom, & Watkins, 2005)
than non-college ADHD adults. Nevertheless, they tend to fall significantly behind persons in
college who do not suffer from ADHD (Barkley, Murphy, & Fischer, 2008; DuPaul et al., 2009;
Heiligenstein, Guenther, Levy, Savino, & Fulwiler, 1999) and those with ADHD have a higher
dropout rate than those without ADHD.
ADHD has been linked to deficits in Executive Functioning (EF) in the literature
(Cortease et al., 2005; Kassubek, Juengling, Ecker, & Landwehrmeyer, 2005; Koechlin, Corrado,
Pietrini, & Grafman, 2000; Lewis, Dove, Robbins, Baker, & Owens, 2004; Monchi, Petrides,
Strafella, Worsley, & Doyon, 2006; Niendam et al., 2012; Stuss & Alexander, 2000; Stuss,
Alexander, Floden, Binns, Levin, & McIntosh, 2002). There is also evidence that EF abilities
are not fully developed until around the third decade. Both established theory and fMRI imaging
support the idea of delayed development (Barkley, 2012). Therefore, it is hypothesized that
there will be a different set of characteristics for the college student population (average age 18-
30) than an adult aged 30 or above.
viii
Given that there are some weaknesses in traditional EF testing (e.g., EF and general
intellectual level is somewhat correlated) (Salthouse; 1996, 2005), the Barkley Deficits in
Executive Functioning Scale (BDEFS) was specifically designed to evaluate EF deficits in adult
individuals with ADHD. This is a new self-report measure identifying functional impairment in
EF abilities with five factors. The goal of this study was to provide further empirical support
regarding the validity and reliability of the Barkley Deficits in Executive Functioning Scale
(BDEFS; Barkley, 2011b). In addition to the five factors, this scale contains an ADHD-EF Index,
which provides an estimate of the likelihood of a diagnosis of adult ADHD (Barkley, 2012b). To
date, this scale has not been investigated for evidence of validity and reliability from an
independent researcher. Additionally, college students have not yet been studied (Barkley,
2011b). Therefore, this study evaluated (a) differences in self- and other-reports on the BDEFS,
(b) the relationship between the BDEFS scales and cognitive functioning, (c) the ability of the
BDEFS to predict ADHD, and (d) the factor structure of the BDEFS with a college student
population.
In this study BDEFS self-reports were collected from 596 college students (with and
without a diagnosis of ADHD). The mean age of the participants was 20.5 years of age and most
demographic variables were consistent with the statistics published for the university where the
data were collected. To evaluate the differences in the self-report form and the other-informant
form of the BDEFS, a Pearson Correlation was conducted comparing self- and other-reports,
using only the sample of students with a diagnosis of ADHD. It was determined that there were
statically significant correlations between the BDEFS-self form and the BDEFS-other form.
These correlations were also statistically significantly different from the correlations that Barkley
found in his original study. In addition, there were statistically significant differences in the
ix
means between the self- and other-informant forms, indicating that college students in general
perceive their symptoms differently, and generally more impaired, than those around them.
Another Pearson Correlation was conducted to determine if the general intelligence of the
participant was related to any of the five factors of the BDEFS. Results indicated that there was
an inverse relationship between intellectual ability and time management skills. As time
management skills decreased, intelligence increased.
When investigating the ADHD-EF index, which is a scale used to predict adult ADHD, a
discriminate analysis was conducted. It was determined that a different set of items was needed
to distinguish college students with and without ADHD than was needed to distinguish these
groups in the adult population. Finally, a confirmatory factor analysis was conducted to see if
the same factor structure of the BDEFS held true for the population of college students. Results
indicated a moderate to good fit for the factor structure in the college student population.
While additional support of validity is needed, the current study did provide additional
evidence for the validity and reliability of the BDEFS. A replication of the newly identified
items of the ADHD-EF Index that are most predictive of adults with ADHD in the college
student population is needed to provide additional support.
1
CHAPTER 1
INTRODUCTION
The goal of this study was to provide further empirical evidence regarding the validity
and reliability of the Barkley Deficits in Executive Functioning Scale (BDEFS; Barkley, 2011b).
The BDEFS is a new self-report measure specifically designed to evaluate executive functioning
(EF) deficits in individuals with Attention Deficit Hyperactivity Disorder (ADHD. Among other
factors, this scale contains an ADHD-EF Index, which provides an estimate of the likelihood of a
diagnosis of adult ADHD (Barkley, 2011b). To date, this scale has not been investigated for
support of validity and reliability from an independent researcher. Additionally, specific
subpopulations (such as college students) have not yet been studied. As part of this validation
process, a college student population was assessed for possible differences in the ways in which
the BDEFS captures information about their executive functioning, along with their likelihood of
having a diagnosis of ADHD. The current study evaluated (a) differences in self- and other-
informant reports on the BDEFS, (b) the relationship between the BDEFS scales and cognitive
functioning, (c) the ability of the BDEFS to predict ADHD, and (d) the factor structure of the
BDEFS. This manuscript proposes that the college student population differs from the general
adult population in terms of ADHD symptoms and EF deficits. A review of the biological
underpinnings of EF and ADHD, current theory, current assessment tools available to measure
EF, and the limitations of these tools is also provided.
Social Significance
Attention-Deficit/Hyperactivity Disorder (ADHD) is generally thought of as a childhood
disorder. It was once believed that the impairing symptoms of ADHD fade in adulthood.
However, research has shown that ADHD often persists past childhood, with similar debilitating
2
symptoms (Antshel et al., 2009; Faraone et al. 2006; Ramirez et al., 1997; Turnock, Rosen, &
Kaminski, 1998). According to results from the United States Comorbidity Replication Study
conducted in 2010, approximately 4.4% of the adult population suffers from ADHD (Keesler et
al., 2010). It is also estimated that 50-65% of children with ADHD will continue to experience
symptoms throughout adulthood (Booksh, Pella, Singh, & Gouvier, 2010; Goldstein, 2002).
ADHD is also one of the fastest growing disability categories on college campuses
(Antshel et al., 2009). College students with ADHD tend to have different characteristics from
their non-college peers who also have ADHD, likely due to differing sets of demands and
stressors. For one thing, they tend to have more protective factors such as previous academic
success and higher cognitive abilities than their non-college peers (DuPaul et al., 2009; Glutting
et al., 2005). Even with the presence of these protective factors, they tend to fall significantly
behind persons in college who do not suffer from ADHD. They are likely to have lower GPAs
and are on academic probation at higher rates than their academic peers (Barkley et al., 2008;
DuPaul et al., 2009; Heiligenstein et al., 1999).
Deficits in EF and symptoms of ADHD are highly problematic in the academic
environment and are correlated with high college drop-out rates and poor academic performance
(Norwalk, Norvilitis, & MacLean, 2009). Given that self-regulation is a requisite skill for
successful navigation of the college environment, a better understanding of the needs of this
college student sub-population must be a top priority. It should be noted that there is a high rate
of stimulant (used to treat ADHD) medication misuse on college campuses, leading to
difficulties in diagnosis due to malingering (Booksh, et al., 2010). Therefore, a closer analysis of
college students as a distinct population from other adults is needed.
3
Biological Factors Explaining the Relationship Between EF and ADHD
The link between EF and ADHD has been discussed in the literature for some time.
There is a large body of evidence from direct neuro-imaging studies that indicates that persons
with ADHD demonstrate patterns of hypoactivation in the areas of the brain responsible for EF
(Cortease et al., 2005). Through a meta-analysis, Cortese and colleagues (2012) provide some of
the strongest support for this assertion. A pattern of hypoarousal was found in children and
adults with ADHD in the areas of the brain responsible for EF (Kassubek et al., 2005; Koechlin
et al., 2000; Lewis et al., 2004; Monchi et al., 2006; Niendam et al., 2012; Stuss & Alexander,
2000; Stuss et al., 2002). This pattern was, however, slightly different for adults than children.
This provides support for the idea that EF development is not complete until later in the lifespan.
The pattern of hypoarousal in the areas of the brain responsible for EF is also supported in the
medical field through pharmacological intervention of ADHD. As Biederman, Spencer and
Wilens (2004) put it, most pharmacological intervention rests upon “the idea that
catecholaminergic hypoactivity in frontal subcortical circuits underlie the disorder” (p. 15). The
stimulant medication reduces this hypoactivity, thus returning the functioning of the EF system
to a normal level, relatively speaking.
Barkley’s Model of EF and ADHD
The biological model of EF and its relationship to ADHD only covers the one narrow
aspect (biology) of this relationship. No consideration had previously been given to variables
such as emotional control, self-awareness, or the social aspect of EF (Barkley, 2012). Given the
lack of breadth in the current biological model, Barkley, one of the prominent researchers in the
field of ADHD and EF, dedicated years of his research to developing a new model of EF and its
relationship to ADHD.
4
Barkley’s new model of EF (Barkley’s Hierarchical Theory of EF) incorporates six levels
of EF and they are arranged in a hierarchical fashion. The six levels are based on the idea of
self-regulation. Impairments in these areas are common in individuals with ADHD.
Many researchers in the field have linked ADHD to deficits of EF (Barkley, 1997, 2011a,
2012; Biederman et al., 2004; Biederman, Spencer, Wilens, Prince, Faraone, 2006; Biederman et
al., 2007; Boonstra et al., 2005; Brown, 2008; Chelune, Ferguson, Koon, & Dickey, 1986;
Grondzinsky, & Diamond, 1992; Fuster, 1997; Hervey, Epstein, & Curry, 2004; Nigg & Casey,
2005; Oosterlaan, Scheres & Sergent, 2005; Seidman et al., 2006; Welsh & Pennington, 1988;
Weyandt, 2009; Willcut et al., 2005). Thus, the controversy is not whether ADHD is associated
with EF deficits. Rather, the controversy is over the way in which they are associated.
Difficulties in measuring EF are at the forefront of this controversy.
Assessing EF: Psychometric Tests or Impairment Rating Scales
At the present time, there are two primary ways of measuring EF: neuropsychological
tests of some type of cognitive ability (psychometric tests) and rating scales of impairment
(questionnaires). Hereafter, these two types will be referred to as EF tests and EF behavior
rating scales, respectively. The results of using EF tests to determine impairment in people with
ADHD across the literature have been inconsistent (Alderman, Burgess, Knight, & Henman,
2003; Anderson, Anderson, Northman, & Mikiewicz, 2002; Barkley & Fisher, 2010; Barkley &
Murphy, 2010; Burgess, Alderman, Evans, Emsile, & Wilson, 1998; Chaytor, Schmitter-
Edgecombe, & Burr, 2006; Vriezen & Pigott, 2002; Wood & Liossi, 2006). Across studies, only
0-10% shared variance has been found between any single EF test and an EF behavior rating
scale (Barkley & Murphy, 2010). This means that an EF test measuring a specific cognitive
5
ability in the lab (e.g., working memory) may not correspond to daily functional impairments
(which are measured by the behavior rating scales).
Given both theoretical and empirical problems with using EF tests, Barkley developed
the Barkley Deficits in Executive Functioning Scale (BDEFS; Barkley, 2011 a, b). The BDEFS
is a behavior rating scale designed to assess functional impairment, rather than specific cognitive
abilities of EF. It is an 89-item self-report scale, which yields five factors: self-management of
time, self-organization/problem solving, self-restraint or inhibition, self-motivation, and self-
regulation of emotion. Additionally, an ADHD-EF Index can also be calculated, predicting the
likelihood of an adult diagnosis of ADHD. Barkley conducted multiple analyses to provide
support for validity and reliability for this new measure. He conducted multiple factor analyses
to provide support for construct validity. Although the final confirmatory factor analysis was
conducted on a pool of 100 items including the final items of the BDEFS, the majority of the
statistical procedures that were conducted used a prototype version of the scale. This prototype
version lacked one of the factors, thus the sub-factor of self-regulation of emotion has less
empirical support (Barkley, 2011b). Also, the analyses were conducted on an adult population
without giving special consideration to the younger portion of this population (i.e., the college
student).
Test Construction
It is important to provide evidence for the validity and reliability of a new measure before
it is used clinically. There are several types of reliability to consider: inter-rater reliability, split-
half reliability, and internal consistency. Reliability refers to the degree of which a measure can
be replicated (Messick, 1989). In this study; however, cross-informant reliability was
investigated. Validity refers to the measurement tool’s ability to actually measure what it
6
purports to measure (Messick, 1989). The clinical consequences of using a measure to
categorize human behavior are so great that it is imperative for researchers to make a concerted
effort to provide support for the validity of the measurement tool to measure the construct at
hand. There are multiple ways to provide support of validity: face-validity, criterion validity,
content validity, and construct validity (Heiman, 2002). The focus for this psychometric study
was construct validity. A factor analysis is one way to provide evidence of construct validity
using theoretical and empirical knowledge (Brown & Cutik, 1993; Brown, 2006). Additionally,
for a measure to be used in discriminating a clinical population from a non-clinical population,
support for its utility is imperative. In the case of the BDEFS, a discriminant analysis is one way
to determine the utility of the ADHD-EF Index. An important feature of a discriminant analysis
is that it provides an accuracy rate for the user in determining group membership (in this case,
ADHD or non-ADHD).
Statement of the Problem
Both established theory and fMRI imaging support the idea that the anatomical areas of
the brain responsible for EFs are not fully developed until an individual’s late 20’s or early 30’s
(Barkley, 2012). Therefore, a college student (average age 18-30) was hypothesized to have a
different pattern of responses than an adult aged 30 or above. The BDEFS is currently normed
by age group, with the youngest age group covering the span 18-35 years old. Based on the
changes occurring in this time period, the current clinical cut-offs and normative data for this
questionnaire might not be appropriate for its use with college students. If this hypothesis is
correct, and if the current clinical cut-offs and normative data are utilized, misdiagnosis might
occur. This, in turn, could lead to inappropriate treatment recommendations and ineffective
treatment. Therefore, the primary goal of this study was to add to the body of literature dealing
7
with the validity, reliability, and normative data of the BDEFS as applied to the college student
population. Several research questions were proposed to this end.
Research Questions
1. What is the relationship between the BDEFS self-report and other-informant report in a
college student population of students with ADHD? What are these relationships on the
following factors: Self-Management to Time, Self-Organization and Problem Solving,
Self-Restraint, Self-Motivation, Self-Regulation, ADHD-EF Index, and Total Executive
Functioning Symptoms? How do these correlations compare to the correlations that
Barkley found in his original study? Are the means of the self-informant reports higher
or lower than the means of the other-informant reports within the college student
population of students with ADHD?
2. In the college student ADHD sample, is there a correlation between intellectual
functioning (as measured by the BIA) and the BDEFS similar to the correlation between
the intellectual functioning and BDEFS in the norming sample? This relationship was
analyzed for the following factors: Self-Management to Time, Self-Organization and
Problem Solving, Self-Restraint, Self-Motivation, Self-Regulation, and Total Executive
Functioning Symptoms.
3. Are the same ADHD-EF Index questions the most predictive of a diagnosis of ADHD in
a college student population as they are in the original normative sample? The current
BDEFS ADHD-EF Index is composed of 11 items. This index was created using a
discriminant analysis to select those items (out of 89) that best discriminate those with
ADHD from a normative sample. Do the same 11 items best discriminate those with
ADHD from a normative sample in a college student population?
8
4. Is the factor structure of the BDEFS the same for college students as it is for the
normative sample, based on a confirmatory factor analysis?
The results of this study may have an impact in the way in which this questionnaire is utilized in
determining the deficits in executive functioning that a college student presents and the
likelihood of a correct diagnosis of adult ADHD and its consequent treatment.
9
CHAPTER 2
LITERATURE REVIEW
This literature review covers multiple aspects of Attention–Deficit/Hyperactivity
Disorder (ADHD) and Executive Functioning (EF). College students with ADHD represent a
distinctive subset of adults with ADHD, and the differences between college students and non-
college student adults are discussed. Specifically, the review focuses on the differences in regard
to ADHD and EF. Next, the review covers the biological and neuropsychological underpinnings
of how EF and ADHD relate to one another. Current measures of EF tests are reviewed,
including their limitations. Although the biological view of ADHD and EF lays the foundation
for further investigation, there is agreement in the field that this model lacks the needed
integration of social influences. Barkley (2012) has provided a theory of EF and ADHD to
address this aspect of the field, and a review of his theory follows. Finally, a review of Barkley’s
(2011b) new measure, The Barkley Deficits in Executive Functioning (BDEFS), is discussed.
An overview of test construction is provided, leading to the current psychometric study of the
BDEFS.
ADHD and How the College Student is Different
ADHD in Adults
Attention-Deficit/Hyperactivity Disorder (ADHD) was classically regarded as a disorder
that affects children, fades throughout adolescence, and virtually disappears in adulthood. As
research has advanced, it has become apparent that ADHD often persists into adulthood, causing
similar impairments across many stages of life (Antshel et al., 2009; Faraone et al., 2006;
Ramirez et al., 1997; Turnock et al., 1998). Results published by the United Stated Comorbidity
Replication Study conducted in 2010 (Kessler et al., 2010) indicated that approximately 4.4% of
10
the adult population displays symptoms of ADHD. Across studies, inattention appears to persist
to a greater extent than the symptoms of impulsivity or hyperactivity (Antshel et al., 2009;
Biederman, Mick, & Faraone, 2000). It is estimated that 50-65% of children with ADHD have
symptoms that persist into adulthood (Booksh, Pella, Singh, & Gouvier, 2010; Goldstein, 2002).
Moreover, DuPaul, Schaughency, Weyandt, Tripp, & Kiesner (2001) reported that there has been
a significant increase in college students reporting ADHD in higher educational settings,
(Antshel et al., 2009).
Prevalence Rates in College Students with ADHD
There have been relatively few studies conducted on the college student population with
ADHD; however, ADHD is one of the fastest growing disability categories on college campuses.
Initial findings by Wolf (2001) suggest that 25% of college students who received disability
services do so under a diagnosis of ADHD. However, this statistic is over a decade old. DuPaul
et al. (2009) indicated that it is hard to get an accurate prevalence rate for college students
because they are not required to disclose their disability at the college level. Based on a
literature review of multiple studies, it appears that 2-8% of college students may have a
diagnosis of ADHD (DuPaul et al., 2009). High ability students (those with high intelligence)
are often not identified as ADHD until college due to their adaptive skills (Booksh, et al., 2010;
Weyandt, Linterman, & Rice, 1995), making the prevalence even more difficult to accurately
report. In addition to the difficulties inherent in identifying college students with ADHD, the
subtypes of ADHD appear to be different in the adult and college populations, compared to the
child population. Heiligenstein and colleagues (1999) indicated that while there is variability
across subtypes in childhood, there is less variability in the adult population. The adult and
11
college populations tend to consist of primarily of the inattentive type or combined type rather
than hyperactive/impulsive type often seen in younger populations.
Academic Functioning in the College Student with ADHD
College students with ADHD are a unique subset of the population of adults with ADHD,
usually ranging in age from 18-30 years old. Although they technically fit into the adult
population, their characteristics are demonstrably different. College students with ADHD are
different from non-college adults with ADHD in several ways. These differences include more
advanced coping skills, past experiences with school success, and higher cognitive abilities
(DuPaul et al., 2009; Glutting et al., 2005). College students may also be burdened with a
different and perhaps higher level of stress stemming from their academic demands (Frazier,
Youngstrom, Glutting, & Watkins, 2007). While college students with ADHD appear to have in
common the aforementioned protective factors, they fall behind their non-ADHD college peers
in many areas. College students with ADHD report more academic problems, have lower GPAs,
and have a higher rate of academic probation than non-ADHD college students (Barkley et al.,
2008; DuPaul et al., 2009; Heiligenstein et al., 1999).
As mentioned previously, there have been relatively few studies on college students with
ADHD. Of the studies published, several have identified barriers that students with ADHD face
that are different than their non-ADHD counterparts. They have difficulty completing tests in a
timely manner, require longer amounts of time to complete assignments, and they have the
perception that they work harder than their non-ADHD peers (DuPaul et al., 2009).
Additionally, Norwalk, Norvilitis, & MacLean (2008) suggested that college students with
ADHD have poorer study habits, a lack of study skills, more difficulties with academic
adjustment (Anstshel et al., 2009; DuPaul et al., 2009), and a lower quality of life (Grenwald-
12
Mayes 2002). Another study by Shae-Zirt, Popali-Lehane, Chaplin, and Bergman (2005)
indicated that college students with ADHD had poorer social skills and lower self-esteem.
College students with ADHD tend to have difficulties with time-management,
organization, concentration, motivation, focusing, test-tasking skills, and study strategies
(Meaux, Green, & Broussard, 2009; Proctor & Prevatt, 2009; Reaser, Prevatt, Petscher, &
Proctor, 2007). They also identify as having poor reading comprehension, sleeping difficulties,
they tend to fail to utilize resources and supports (Meaux et al., 2009), and report more anxiety
(Proctor & Prevatt, 2009; Reaser, et al., 2007). College students with ADHD must learn to
navigate a variable schedule with no structure and an overwhelming number of campus activities
to choose from. These complexities were likely managed by their parents prior to college
(Meaux et al., 2009) resulting in a lack of developed skills to cope with these challenges. Given
these difficulties, it is not unexpected to see a higher school failure rate in college students with
ADHD (Antshel et al., 2009; Barkley et al., 2006; Biederman et al., 1993, 2006).
The college environment brings about many additional struggles in a person with ADHD
than is seen in the non-college environment. Due to this, college students with ADHD are
thought to struggle even more due to deficits in their executive functioning (EF) abilities
(Boonstra, Oosterlaan, Sergent, & Buitelaar, 2005). College students with ADHD have
significantly poorer EF ability, and EF problems are thought to directly affect performance in the
academic world (Nadeau, 2005). This includes problems focusing, making deadlines, task
completion, and sustaining effort in presumed irrelevant tasks (Murphy, 2005; Proctor & Prevatt,
2009). Additional skills, including self-organization and goal-setting, fall under the realm of EF
and are generally impaired in the ADHD population (Wolf, 2001).
13
Diagnostic Issues
ADHD is currently defined by the Diagnostic and Statistical Manual of Mental Disorders,
Fifth Edition (DSM-5) as “a persistent pattern of inattention and/or hyperactivity-impulsivity that
is more frequently displayed and more severe than is typically observed in individuals at a
comparable level of development” (APA, 2013, p. 59). There are three subtypes of ADHD:
ADHD Predominantly Inattentive Type, ADHD Predominantly Hyperactive-Impulsive Type,
and Combined Type. An individual must have six symptoms of inattention or six symptoms
hyperactivity-impulsivity to be classified as predominately that particular type. If an individual
has a minimum of six of each type, then they are considered to be combined type. For adults, the
minimum threshold of symptoms is five, rather than six in the new version of the DSM. These
symptoms must persist for at least six months and must cause impairment across multiple
settings, which are not consistent with their developmental level. Even though these are the
official diagnostic criteria, there is considerable debate in the field whether this set of standards
captures the essence of this disorder. As will be discussed, many top researchers in the field
have attributed several of the impairments associated with ADHD to be deficits in EF. There is
some evidence that the severity of symptoms of ADHD is associated with deficits as measured
by EF tests (Barkley et al., 2008; Jonsdottier et al., 2006; Stavro, Ettenhofer, & Nigg, 2007) and
EF behavior rating scales (Barkley, 2011c; Barkley & Murphy, 2010).
Diagnosing ADHD in the college student and adult population is controversial (Booksh et
al., 2010). In general, the diagnostic criteria for ADHD in all but the most recent version of the
DSM in 2013, were designed with children in mind, which makes accurate diagnosis particularly
challenging. One reason that has been suggested for this challenge is that the threshold to meet
the criteria for ADHD may not have the sensitivity needed to accurately diagnose a college
14
student or adult (Barkley, 1996; Heiligenstein et al., 1999). The age-appropriateness of the
symptoms designated in the DSM may explain some of the reason for this. Therefore, the
estimates of ADHD in adults and college students may be an underestimate. For example the
symptom “leaves seat when remaining seated is expected” is far less likely in an adult who is
capable of remaining seated for social reasons. However, the same internal struggle of
restlessness may exist and cause significant impairment. An additional difficulty in the accurate
diagnosis of college students with ADHD is the fact that in some cases, impairment does not
arise academically until the student reaches the college setting. One stipulation in the diagnostic
criteria is that some symptoms must be present as young as age twelve (previously age seven in
prior versions of the DSM). Academically advanced students may not be able to accurately
identify problems prior to that age (Booksh et al., 2010). However, research has shown that
there is no effective difference in symptoms and impairment between students who show early
symptoms and students whose symptoms do not present as impairments until later in life
(Faraone et al., 2006). Therefore, it is conceivable that an accurate diagnosis of the college
student population may not be feasible.
An additional concern in the diagnosis of college students with ADHD is that of
secondary gains. Given the rampant abuse of stimulant medications, some students do attempt to
feign a diagnosis of ADHD to gain access to cognitively enhancing medication (Booksh, to al.
2010). The stimulant medication used to treat ADHD also increases the ability to maintain
attention in the non-ADHD population (Rapoport, Buchsbaum, Zahn, Ludlow, & Mikkelsen,
1978), which can encourage misuse. The literature suggests that there has been an increase in
the misuse of simulant medication in the university setting (McCabe, Knight, Teter, & Wechsler,
2005; Teter et al., 2005; Tuttle, Sheurich, & Ranseen, 2007; White, Becker-Blease, & Grace-
15
Bishop, 2006). Incentives other than medication for ADHD malingering exist as well. With the
Rehabilitation Act of 1983, students with disabilities are entitled to the provision of
accommodations in the academic setting (Gordon & Keiser, 1998). The greatest incentive is the
accommodation for extended test-taking time, which can include high-stakes testing
accommodations. Information about ADHD is readily available on the internet, which may
increase the likelihood that a student will feign their symptoms to receive a diagnosis of ADHD.
Given these issues, effective assessment tools designed for the college population are needed.
The Relationship Between ADHD and EF: How Biology and Neuropsychology Inform Our
Understanding
The following section briefly discusses the biological and neuropsychology literature
which links ADHD and EF. The biological underpinnings of ADHD and how this is linked to
EF are addressed. Additionally, a brief synopsis of pharmacological treatment for ADHD is
discussed, especially as that informs our knowledge of EF.
There is a robust body of evidence from direct neuro-imaging studies which indicates
that persons with ADHD demonstrate patterns of hypoactivation in the areas of the brain
responsible for EF. A meta-analysis of 55 fMRI studies provides some of the strongest support
for this assertion (Cortese et al., 2012). In adults and children, Cortese and colleagues found a
consistent pattern of hypoarousal in areas of the brain responsible for EF (Kassubek et al., 2005;
Koechlin et al., 2000; Lewis et al., 2004; Monchi et al., 2006; Niendam et.al. 2012; Stuss &
Alexander, 2000; Stuss et al., 2002). These areas were specifically observed during EF tasks in
persons with ADHD, and the results of this meta-analysis confirms that the current body of
literature describes direct evidence of a differential level of activation in these areas of EF for
persons with ADHD vs. normal adults and children. It is notable that for adults in the Cortese et
16
al., study, the areas of hypoactivation were overwhelmingly (97%) associated with the
frontoparietal network, which is the area directly associated with EF. Children, on the other
hand, demonstrated similar rates of hypoactivation in ventral attention (44%) and frontoparietal
(39%) networks. This suggests that the neurobiology of adults and children with ADHD may
differ in measurable ways. The difference in age is an important point given that Barkley’s
theory (described in the following sections) proposes that EF abilities are not fully developed
until around age thirty (Barkley, 2012).
In addition to evidence from imaging studies, there has been a parallel course for medical
treatment of ADHD and areas of EF, particularly the ability to sustain attention.
Pharmacological interventions intended to improve sustained attention have been associated with
ADHD. Most pharmacological intervention rests upon, as Biederman, Spencer and Wilens
(2004) put it, “the idea that catecholaminergic hypoactivity in frontal subcortical circuits underlie
the disorder (p. 14).” Essentially, the individual with ADHD experiences lower cortical arousal
in prefrontal areas. The regions with the greatest role in attention, impulse control, and planned
behavior remain in a sub-optimal range. Despite discussion regarding the functions of specific
transmitters implicated in the reward pathways (see Gonon, 2008 for a detailed analysis),
suboptimal cortical arousal remains the primary explanatory paradigm. In other words, in the
same areas that persons with EF deficits due to injury have lesions, persons with ADHD display
a natural tendency toward hypo-activation. Thus, ADHD can be seen as a cluster of long-term,
naturally derived set of deficits in executive functioning due to suboptimal activation (rather than
damage) in key portions of the brain.
The theory of suboptimal arousal in ADHD (as described above) was most eloquently
and memorably laid out by Zentall & Zentall (1983). Although this preceded the current
17
diagnostic understanding of ADHD, they asserted that hyperactive behavior “represents a
functional set of homeostatic responses to conditions of abnormal sensory input. Attempts to
correct chronic imbalances in arousal through antecedent manipulations of chemical and sensory
stimulation have been relatively successful…” (pp. 446). In essence, hyperactive behavior is not
a direct symptom of ADHD. Rather, it represents an attempt on the person’s part to manipulate
or interact with their environment in such a way as to increase the amount of stimulation
available. This is done because (as noted above) the systems responsible for maintaining
vigilance and sustaining goal-directed behavior are receiving a sub-optimal level of stimulation.
This has been well established by assessing working memory (Rottschy et. al., 2012), and long-
term EEG patterns in ADHD (Doehnert, Brandeis, Schneider, Drechsler, and Steinhausen, 2012;
Loo, et al., 2009). The electrophysiological responses indicative of preparation or attention are
markedly suppressed in groups of persons with ADHD vs. normal populations.
A proper understanding of the neurobiology of ADHD and associated deficits in
executive functioning forms an important foundation for the comprehensive diagnosis and
treatment of ADHD. A common misunderstanding about ADHD in clinical practice is due to
the variability in symptoms. People with ADHD can focus without difficulty on activities they
find salient or when they are avoiding a negative consequence (Baxter & Murray, 2002). There
are times that they may be relatively unimpaired by their ADHD. This apparent ability to
selectively attend to some tasks and failing to attend to others is likely what led the general
public to assume that people with ADHD just lack willpower. According to Brown (2008),
however, the cause of the situational variability is essentially chemical. When confronted with a
task that is personally appealing or frightening because of consequences, the brain instantly
provides chemical stimulation via neurotransmitters to activate relevant EF (Brown, 2008).
18
Baxter & Murray (2002) suggest that this is not subject to voluntary control. The results here tie
in with the aforementioned body of research from Zentall and Zentall (1983), which is a
theoretical framework and empirical support for the perspective of ADHD as a deficit in
physiological arousal, rather than attention.
The suboptimal arousal model indicates a suboptimal natural level of arousal in the areas
responsible for focusing attention in the individual with ADHD. When internal and external
stimulation are insufficient, the individual attempts to regulate their own levels by changing
activities, attentional focus, or experience (Zentall, 2005). This is an important distinction
between persons with ADHD and persons with lesions in areas of the brain responsible for poor
executive functioning: For persons with brain damage, performance on measures of ability in
impacted areas may be expected to depend primarily upon the nature of the task. For persons
with ADHD, the context and level of arousal may be just as important as the actual criteria of the
task. Thus, there is an implication that although on average persons with ADHD can be
expected to display dysfunctional EF, performance on a single discrete task may not be impaired.
This poses a number of psychometric issues with EF tests, which is addressed later in the
manuscript.
Limitations of the Biology Based Model
Although the preceding biological model of EF as it relates to ADHD incorporates the
building blocks for future models of this relationship, the current model falls short of
conceptualizing the larger picture. The model does not take into consideration ideas of
emotional control, self-awareness, or the social aspect of EF (Barkley, 2012). Impairments in
emotional control, occupational difficulties, and moral difficulties are absent from the biological
model. However, such deficits are readily apparent in patients with injuries to the areas of the
19
brain responsible for EF. Barkley postulated that social facets of EF dysfunction are overlooked
because they are not as readily observed in a laboratory setting (measured by EF tests).
Barkley’s interpretation is not without precedent. Eslinger (1996) viewed EF as being
comprised of “social executors,” which serve the social functions. This is one of the only models
of EF to include discrete social components. These components are: social regulations, social
self-awareness, social sensitivity, and social salience. He considered social problems “the most
distinctive feature” of EF. However, there was no truly comprehensive model that addressed
social functioning of EF, and how it relates to ADHD, until Barkley (2012) promulgated his
theory of EF.
Barkley’s Hierarchical Theory of EF
Despite the ways (described in the preceding section) in which his theory differs from the
classical neuropsychological perspective, Barkley’s view of EF is anchored in self-regulation:
“EF is the use of self-directed actions (self-regulation) to choose goals, and to select,
enact, and sustain actions across time towards those goals, usually in the context of others
and often relying on social and cultural means. This is done for the maximization of
one’s long-term welfare as the person defines that to be” (Barkley, 2012 p. 167).
From a functional perspective, EF is “a form of self-directed action aimed at modifying
one’s behavior so as to make a future goal, end, or outcome more or less likely to occur.”
(p. 167).
Each component of EF represents a particular type of self-regulation (Barkley, 1997a,
1997b). In a variation on this then, Barkley refers to EF behaviors as “those self-directed actions
needed to choose goals and to create, enact, and sustain toward those goals (p.168),” meaning
that EF equals self-regulation (Barkley, 2012). This is not a novel conclusion. Wagner and
20
Heatherington (2011) and Eslinger (1996) found this to be the most common feature of EF in
neuropsychology. The view of EF as self-regulation comes up frequently in the role of working
memory, which is one of the many cognitive abilities of EF (Hoffman, Friese, Schmeichel, &
Baddeley, 2011). Self-regulation is also described in the linkage of attention networks
specifically, and EF more generally (Rueda, Posner, & Rothbard, 2011).
If we accept Barkley’s proposition that EF is a means for goal-directed action, a
definition of “goals” and “means” must be provided. Goals are understood to be things in the
future that provide a relative reduction in dissatisfaction compared to the individual’s present
state (Barkley, 2012; Mises, 1990). The actions resulting in attainment of the goal are the means
to that end. Goals are completely subjective and personal. This means that methods for
reduction of dissatisfaction are personal, and cannot be judged as good or bad from an external
perspective. Thus, this type of goal cannot be assumed for the sake of empirical measurement.
Furthermore, if two people make a decision using the same information, they may well come up
with differing (yet viable) solutions unique to the goals and values they have (Barkley, 2011a;
2012).
Barkley’s theory of EF is built on the idea of the extended phenotype. This comprises
four levels of EF, each incorporating a set of pre-executive levels (Barkley, 2012). The extended
phenotype is a biological concept introduced by Dawkins (1982). Essentially, a phenotype
extends beyond the organism into its environment, incorporating all effects that the gene has on
its environment. Dawkins considered the extended phenotype a critical component of the
evolutionary theory (1982). Barkley’s hierarchical model, along with implications for the
manner in which EF impacts phenotypic expression, has been described in the following
sections.
21
The pre-executive level comprises the central-nervous system (CNS). This is an
integrated system with a high degree of automaticity. Included are attention, memory, spatial
and motor functions, primary emotions and motivations. Pre-executive functions include
behaviors such as automatic responses and operant conditioning. Operant conditioning occurs
when animals learn, which allows rapid adaptation to the environment rather than relying on the
laborious evolutionary process (Barkley, 2012). Operant conditioning is not EF, but is a
prerequisite ability.
The first EF level is the instrumental-self-directed level (Barkley, 2012). The basic
neuropsychological functions at the pre-executive level take on a self-directed component in the
initial level of EF. According to Barkley, these include: self-directed attention (self-awareness),
inhibition (self-restraint), self-directed sensory-motor actions (nonverbal working memory;
imagination), self-directed private speech (verbal working memory; verbal thought), self-
directed appraisal (emotion-motivation), and self-directed play (innovation, problem solving).
Early in the organism’s development, these processes move from observable behaviors to
opaque, internalized functions. These are the underlying components of goal-attainment. The
title of “instrumental” is derived from the observation that these components are used to achieve
goals but are not directly capable of attaining the goals (Barkley, 2012).
The methodical- self-reliant level includes observable goal-directed actions and
behaviors (Barkley, 2012). A brief sequence of actions leading to a goal is termed a method (a
recipe). In this level, a single step is insufficient to attain a goal, necessitating the chain of
action. This level is typified by short or near-term goals. There is a component of self-reliance
because the goals are general oriented towards independence from defense against others. This
level is associated with executive behavior rather than executive cognition (Barkley, 2012).
22
In the tactical –reciprocal level, the concept of social symbiosis is introduced into the
EF system. At the methodical level, independence and self-defense are privileged. In this level,
others are used to obtain goals of mutual benefit, in what Barkley describes as a socially
symbiotic relationship. The word tactical refers to linking longer-term actions together to
achieve a higher order goal (Barkley, 2012).
The strategic-cooperative level refers to the hierarchical organization of tactics leading
to multifaceted, multi-stage behavioral patterns over an extended timeframe. This allows for
larger-scale, longer-term goals to be planned for. The hierarchy now consists of methods, nested
under tactics, which are nested under strategies that eventually attain goals at a substantial
distance in time. The strategic level shifts “cooperative” activities to the fore by relying on
higher order social engagement in attaining goals exceeding the ability of an individual (Barkley,
2012).
The first four levels of EF suffice in describing most human behavior. However, Barkley
argues that an additional level at the pinnacle of the hierarchy can be added. He believes that
some civilizations have reached the principled-mutualistic stage. In theory, this represents the
broadest community or social ecology in which most persons live. This level synthesizes sets of
strategies with an eye towards the most expansive and long-term aspects of human goal-seeking.
It applies principled behavior to a community setting in order to simultaneously attain long-term
self-interests and mutualistic community goals (Barkley, 2012).
Barkley (2012) describes the sequential development of each level as being directly
related to the increase in the prefrontal cortex capacities, primarily involving future-oriented
thought. Barkley indicated that through theory and fMRI imaging, individual’s anatomical areas
of the brain responsible for EF functioning are not fully developed until an individual’s late 20’s
23
or early 30’s (Barkley, 2012). Since the maturation process is not fully developed in the EF
system until approximately 30 years, it has been suggested the essential point in Barkley’s model
versus most previous models is the social component. Barkley’s (2012) model is tied heavily to
the concepts of a behavioral impairment, rather than focusing on specific brain function. As
such, his theory would be difficult to measure with EF tests (psychometric) alone, and can best
be assessed by functional outcomes (e.g. impairments). As will be discussed in a later section,
Barkley has developed a behavior rating scale to assess impairment in EF function, reducing the
need for EF test, which have been the gold standard in the field.
In assessing damage to the prefrontal cortex, EF functions can be classified along eight
parameters whose development is tied to the maturation of the prefrontal cortex. In his
explanation, Barkley (2012) identifies the eight characteristics as separate entities; however, he
admits that when the EF system is fully mature, they are interrelated. These characteristics are
spatial capacity, temporal capacity, motivational capacity, inhibitory capacity, conceptual/
abstract capacity, behavioral-structure capacity, social capacity, and cultural capacity. In spatial
capacity, the “individuals come to purposefully rearrange or organize their surrounding physical
environment to assist in goal attainment” (Barkley, 2012 p. 74). This requires an ability to
mentally represent spatial distances. The ability to “reflect how far into the future an individual is
capable of contemplating a goal” (Barkley, 2012 p. 74) is temporal capacity. Sometimes this is
referred to as the “time horizon.” This refers to the time between an event and the preparation
for that event. The preceding capacities are required as a basis of the motivational capacity,
“which is distinguished by the fact that this capacity reflects an appraisal of that future, while the
spatial and temporal capacities above are comprised of cognitive parameters or purely
informational” (Barkley, 2012 p 75). The motivation comes from the personal value placed on
24
the delayed goal or outcome that is given by the individual. Oftentimes, attainment of the goal
relies on how strongly the individual feels about the goal. The “extent to which and the duration
over which an individual must inhibit their responses to prepotent events, restrain their actions,
and otherwise subordinate their immediate interests for the sale of the goal” (Barkley, 2012 p.
76) is considered inhibitory capacity. Individuals who are unable to do this are often classified
as irrational, impulsive, or selfish. Conceptual/abstract capacity is “the degree of abstractness of
any rules that are being considered or followed to attain a goal” (Barkley, 2012 p 76). This can
be simple rules such as “stop” and can be more complex such as “do onto others as you would
have done onto you.” Behavioral-structure capacity “is a capacity for structuring increasingly
complex, hierarchically organized and appropriately sequenced actions towards goals” (Barkley,
2012 .p. 76). This sequenced action is represented in each of the levels of EF as indicated by the
first term, from the instrumental capacity to the principled capacity. The last two capacities
represent the social environment discussed above. Social capacity refers “to the number of other
individuals (and eventually social networks) with which the individual must interact, reciprocate,
and cooperate so as to effectively attain the goal being contemplated” (Barkley, 2012 p. 76).
Cultural capacity “refers to the degree of cultural information and devices (methods, inventions,
products, etc.) or scaffolding that the individual is adopting to attain the goal under
contemplation (Barkley, 2012 p. 77).
In summary, Barkley’s hierarchical model focuses on self-directed behavior and the
ability to utilize self-regulation. The hierarchical levels begin at the pre-executive level and
progresses to the most advanced level of the principled-mutualistic stage, which represents the
most socially and culturally bound level. Barkley then looks at eight developmental parameters,
which are more behavioral in nature and reflect impairments found in individuals with prefrontal
25
cortex damage. This includes motivational capacity, temporal capacity, and inhibitory capacity,
among others. Barkley has also added a sociocultural aspect to EF, which was lacking from
previous models and definitions. Additionally, Barkley’s model operationalizes the definition of
EF, allowing the concept to be empirically measured. As will be discussed in a later section,
Barkley has also developed a rating scale to assess for deficits in EF based on this model
(Barkley, 2011b).
EF Deficits as Manifested in ADHD
Background
In preceding sections, EF has been described largely from a biological or theoretical
(rather than clinical) perspective. However, several other clinical diagnoses are often associated
with deficits in EF. As was previously noted, brain damage (such as TBI or stroke) (Levin &
Hanten, 2005; Weyandt, 2009) often results in acquired deficits in EF. Some mental illnesses
(especially mood disorders) (Murphy, Barkley, & Tracie, 2001; Thompson et al., 2009) are also
associated with fluctuations in EF during acute phases. However, few diagnoses share the same
degree of overlap with EF deficits as ADHD. Over the years, multiple researchers in the field
have linked ADHD to deficits of EF (Barkley, 1997, 2011a, 2012; Biederman et al., 2004;
Biederman et al., 2006; Biederman et al., 2007; Boonstra et al., 2005; Brown, 2008; Chelune et
al., 1986; Grondzinsky, & Diamond, 1992; Fuster, 1997; Hervey et al., 2004; Nigg & Casey,
2005; Oosterlaan, Scheres & Sergent, 2005; Seidman et al., 2006; Welsh & Pennington, 1988;
Weyandt, 2009; Willcut et al., 2005). Based on this sample of current literature, the controversy
is not about whether ADHD is associated with EF deficits. Rather, it hinges on the way in which
they are associated. As previously discussed, there is sufficient evidence from the biological
perspective that ADHD and EF are related. Some researchers outside the field of biological
26
neuroscience indicate that there is overlap between ADHD symptoms and EF deficits as
determined by EF tests. Others, such as Barkley (2011c), have gone as far as to say that ADHD
and EF deficits are synonymous, that ADHD can be called Executive Function Deficit Disorder.
Barkley’s Theory of EF and its Relationship to ADHD
Barkley (2012) has presented, to date, one of the most comprehensive theories on the EF
as it relates to ADHD described above. Based on his Hierarchical Theory of EF, Barkley has
mapped the symptoms and impairments of ADHD onto levels in his theory of EF. As a review,
Barkley proposed a hierarchical theory of EF including six levels; pre-executive,
instrumental/self-directed, methodical/self-restraint, tactical/reciprocal, strategic/cooperative,
extended/utilitarian. He then maps the components of ADHD impairments onto these levels.
Again, Barkley has defined EF as:
“the use of self-directed actions (self-regulation to choose goals and to select, enact, and
sustain actions across time toward those goals usually in the context of others often
relying on social and cultural means. This is done for maximization of one’s long-term
welfare as the person defines that to be” (Barkley, 2011a p.167).
Barkley (2012) asserts in his theory that there are three brain networks associated with
ADHD and EF deficit. 1) The frontal–striatal circuit, which is related to deficits in response
suppression, inhibition, working memory, organization and planning. Barkley has dubbed this
the “what” system of EF. 2) The frontal-cerebellar circuit which is connected to motor
coordination deficits and difficulties with timeliness of behavior. This is recognized as the
“when” of EF network. 3) The frontal-limbic circuit is related to the systems of emotional
control, motivation deficits, hyperactivity/impulsivity, and low frustration tolerance (aggression),
or the “why” in Barkley’s model. Given that all of these brain networks of EF are readily
27
identified as ADHD symptoms, Barkley (2011b, c) has asserted that ADHD essentially equals
Executive Functioning Deficit Disorder.
Barkley (2011b) then supports this view with empirical research. How does ADHD fit
with EF? A factor analysis of ADHD symptoms (using symptoms checklists) and EF symptoms
(using a behavior rating scale) was conducted (Barkley, 2011a), resulting in one overarching
factor, similar to g in intelligence theory, which supports his claim that ADHD is Executive
Functioning Deficit Disorder.
Barkley (2011b, c) has indicated that he does not agree with the very name of ADHD
(attention deficit) because it is essentially useless as a diagnostic feature. Similarly, Zentall
(1983; 2005) argued that attention deficit was a misnomer. Many disorders, (depression, bipolar,
anxiety, schizophrenia) (Murphy, Barkley, & Tracie, 2001; Thompson et al., 2009) have
inattention as a symptom. Barkley narrowed down the concept of inattention in ADHD
specifically and defined it as inattention towards a task or outcome, future goal, which must be
organized and monitored over time. He suggests that working memory deficits are the reason for
this inability to remain focused on a future goal, and the diagnostic criteria of “changing from
activity to activity or not completing a task.” Again, when symptoms of ADHD and EF
symptoms were subjected to a factor analysis, one large factor was evident. If you add the
results from EF tests, it does not change the factor structure. Essentially, Barkley asserts, ADHD
is Executive Functioning Deficit Disorder. Barkley, however, does not appear to address the
issue of other deficits or disorders that also have deficient or damaged prefrontal cortex. For
example, a patient who is post cardio vascular accident generally has deficits in EF. It is not
addressed whether this would also be classified under the same umbrella of Executive
Functioning Deficit Disorder, as ADHD is in Barkley’s model. Barkley (2011b, c) claims all
28
ADHD is Executive Functioning Deficit Disorder, but is all Executive Functioning Deficit
Disorder also ADHD?
Barkley (2011b, c) indicates that severe prefrontal cortex injuries often cause significant
impairment in all levels of the EF hierarchy, including at the lower levels. This may be related to
the fact that EF impairments in individuals with traumatic brain injuries tend to show up on EF
tests more consistently than in individuals with ADHD. However, prefrontal cortex disorders
tend to show greater impairment in the highest two levels of EF, as presented by this model. EF
disrupts all five levels of EF/Self-Regulation. It is particularly troublesome in the tactical level
and higher levels, which in turn generates difficulties of self-regulation across time. The ability
to hierarchically organize behavior, anticipate forthcoming events, and maintain a long-term goal
direction is affected. Barkley suggests that it is not an Attention deficit disorder, but an Intention
deficit disorder (attention to mental events and the future). ADHD is a deficit in performance,
not knowledge (Barkley, 2011b).
The diagnosis of ADHD, especially in adults, is somewhat complex. There is no
biological or neurological test that can confirm a diagnosis. This introduces error into clinical
practice and research. A problem in the field arises when researchers attempt to evaluate the
relationship between ADHD and EF deficits using EF tests, which have been the “gold standard”
for the past several decades when assessing EF (Barkley, 2012). These tests measure cognitive
abilities such as working memory, inhibition, and attention. One criticism of EF tests is that they
measure a single point in time and do not require the subject to self-regulate behavior over time
(Clark, Prior, & Kinsella, 2000). The following sections will describe some of the current EF
tests that evaluate cognitive abilities, explain why they are controversial, and then describe
measurement of EF via behavior rating scales.
29
Assessing EF
There are a number of EF tests; however, there are also a number of inherent problems
with assessing the construct. Following is a discussion about some of the current available tests
of EF functioning, and by extension, measures of ADHD. One problem with assessing EF is that
there is substantial evidence supporting the view that, at least as assessed by common measures,
EF and intelligence (IQ) are related (Jung, Yeo, Chiulli, Sibbitt, & Brooks, 2000).
Certain researchers have gone so far as to suggest that EF and IQ represent essentially the
same thing (Obonsawin et al., 2002). However, this is far from a majority view. Some lines of
research indicate that EF is mediated by IQ (Antshel et al., 2010). Others say they are only
somewhat related, but are not the same thing (Ardila, Ostrosky-Solis, Rosselli, and Gomez,
2000; Miyake, Friedman, Emerson, Witzki, & Howerter, 2000). At the present time, there are
two primary ways of measuring EF: neuropsychological tests (psychometric) and rating scales
(questionnaires). Psychometric neuropsychological tests (or EF test) are defined as an objective
measure of an ability to perform a task, e. g., speed at which a task is completed or number of
digits repeated backwards. An EF behavior rating scale is a measure of subjective belief in a
person’s ability to accomplish a task and tends to focus on perceived impairment, e.g., a
questionnaire asking about motivation or frustration tolerance. There is a long-standing history
of utilizing EF tests in assisting in the diagnosis of ADHD; however, the use of EF behavior
rating scales of impairment is relatively new.
Given the many cognitive abilities associated with EF, there are multiple ways in which
researchers approach measuring the construct. Some EF tests are designed to assess specific and
isolated abilities, such as working memory or inhibition. Others are grouped into test batteries
which purportedly measure “executive functioning.”
30
EF Tests
A large variety of assessment tools exist, and a complete review is outside the scope of
this literature review. Thus, only representative measures and batteries will be discussed, and
only as they relate to the EF abilities which they were intended (or are commonly used) to
measure.
Working memory. Working memory is arguably the best studied construct of EF.
Several common working memory tests are: The Simon Game, testing non-verbal working
memory (Lezak, Howieson, & Loring, 2004); Digit Span and Arithmetic, subtests of the
Wechsler Adult Intelligence Scale (WAIS –IV; Wechsler, 2009); and the Six Element Task
(SET; Burgess & Shallice, 1996). Generally these EF tests measure working memory by
incorporating components of straight memorization and a manipulation (or interference) task.
For example, in the Digit Span subtest of the WAIS-IV, a participant repeats a string of numbers,
and then repeats them backwards. Finally, the numbers are recited in sequential order.
Inhibition. To assess the construct of inhibition, common EF tests include: The Stroop
Color Word Test (Stroop, 1935; Trenerry, Crosson, Deboe, & Leber, 1989), the Hayling
Sentence Completion Test (HSCT; Burgess & Shallice, 1996, 1997), and the Wisconsin Card
Sort (WCST; Heaton, 1981). As is common on this type of test, they measure the capability of
inhibiting opposing responses with the existence of salient and contradictory stimuli. Although
the WCST requires the ability to inhibit previous response patterns (which will increase the
number of perseverative responses), it also requires several other EF abilities. These include
planning, attention, set shifting, and working memory. The Color Stroop Test and the WCST are
the two most common tests in this category, with the Color Stroop Test being the most common
measure which is primarily an inhibitory task.
31
Problem solving and set-shifting. Problem solving and set-shifting are additional
constructs often assessed with EF tests. Common tests used are WSCT; Tower of London
(Shallie & Burgess, 1996), also known as the Tower of Hanoi; Trail Making Tests (TMT-A,
TMT-B; Reitan & Wolfson, 1985); and Cognitive Estimations Test (CET; Shallice & Evans,
1978). Other than the WSCT, discussed above, the Trail Making Test is the most notable. This
test purports to measure visual search, scanning, processing speed, and mental flexibility (set-
shifting). There were relatively few norms for this set of tests in its earlier years, despite its
common use (Matarazzo et al., 1974; Reitan, 1959). However, this subtest has been incorporated
into the Halstead-Reitan Neuropsychological Battery, which will be discussed later, and now
includes the ability to norm the scores appropriately.
Vigilance/attention. Vigilance, also referred to as attention, is another common
construct with several EF tests available to measure it. These are also used heavily in the
evaluation of ADHD, which is discussed in more detail in a following section. There are many
tests of vigilance or attention, but some of the most notable are the: Digit Vigilance Test (Lewis,
Kelland, & Kupke, 1990), Test of Variables of Attention (TOVA; Greenberg, 1991), Conners’s
Continuous Performance Test (CPT; Conners, 1995), and Gordon Diagnostic System (Gordon,
1983). Many of these tests work on the same basic principle as the CPT, and often incorporate
computerized administration. A single character is shown on the screen at a rate of
approximately one per second. The participant is to click a button when a specific character
(known in advance) is displayed. The total time of the test varies from a few minutes to
approximately 12 minutes. The relatively brief administration times (compared to more
research-oriented measures of vigilance in fields such as human factors) are likely chosen for
practical rather than psychometric reasons. Generally speaking, a score for total errors,
32
commission errors, and omission errors is given. The task requires the participant to remain
vigilant for extended lengths of time. The research has shown an increase in false positives with
a CPT type test (Fischer, Newby, & Gordon, 1995). Most of the research in this area is on
children with ADHD. In children with ADHD, a study showed that there was little ability for the
Conner’s CPT to discriminate between ADHD children and a clinical control group (McGee,
Clark, & Symons, 2000). A different study conducted by Forbes (1998), found that the TOVA
had a greater ability to predict ADHD children from control group children than the CPT did.
However, these types of studies have been plagued by methodological problems.
EF test batteries. In addition to EF tests that measure single constructs (such as the
CPT), several full batteries of neuropsychological tests (EF test batteries) have been developed to
measure a range of EF abilities. The most notable of these batteries is the Halstead-Reitan
Neuropsychological Test Battery (Reitan & Wolfson, 1985); Delis-Kaplan Executive
Functioning System (D-KEFS; Delis, Kaplan, & Kramer, 2001); and the NESPY-2 (Korkman,
Kirk, & Kemp, 1998), a developmental neuropsychological assessment for children. In addition
to these batteries, a single test may capture several aspects of EF. For example, the WCST is
also a single test design which purports to measure several constructs of EF collaboratively.
Similarly, the Rey-Osterrieth Complex Figure (Rey, 1941) is a single measure requiring the
ability to utilize several abilities.
The lack of a set of neuropsychological measures that had cohesive norming data was
once a primary concern in the clinical field of neuropsychology. The above mentioned test
batteries were developed by combining many of the commonly used single construct tests into a
battery, which was then normed at one time (Homack, Lee, & Ricco, 2005). These batteries
include subtests covering all of the constructs individually and in collaboration with each other.
33
Despite this, controversy remains active in the research community regarding these full-battery
tests of EF, especially about their reliability and validity (Schmidt, 2003). Additionally, research
and psychometric information on these tests often does not include a discussion of clinical
utility. The tests purport to measure a specific deficit; however, there is little information on
how that deficit affects the individual with regards to specific areas of day to day impairment.
Limitations of EF Tests
Disagreement exists regarding the best way to measure EF. Proponents of EF tests
measuring a single construct would state that EF represents multiple, separate processes working
together. However, the fact that few (if any) of these tests can measure purely one process lends
credence to the idea that EF is NOT a single ability, but a group of abilities working together.
One could not use working memory without first maintaining attention to the stimulus data. One
could not set-shift or problem solve without the ability to inhibit interference, use working
memory to maintain the problem to be solved, or maintain attention to the problem as it is
expressed. Although these processes are laid out as separate abilities, it is impossible to use one
without employing another.
It is noteworthy that the construct of IQ has been largely absent from the equation. Yet
Salthouse (1996, 2005), hypothesized that the underlying cognitive abilities of perceptual speed
and reasoning are factors related to all EFs. There has been some research to support this idea
(Duncan Emslie, Williams, Johnson, & Freer, 1996). Salthouse (2005) noticed that performance
of the WCST was related to reasoning ability and perceptual speed. In fact, the original manual
for the Gordon Diagnostic System (a relatively straightforward continuous performance task)
notes a relationship between the processes that the Gordon measures and intelligence.
34
In decades of research, only a minority of adults with prefrontal cortex damage or ADHD
consistently perform in the impaired range across all these tests of specific abilities or combined
batteries (Alderman, Burgess, Knight, & Henman, 2003; Vriezen & Pigott, 2002). The results of
EF test batteries, according to Barkley and colleagues, had no or low relationship to the
impairments evident in their daily life as measured by behavior rating scales, suggesting low
ecological validity (Anderson et al., 2002; Barkley & Fisher, 2010; Barkley & Murphy, 2010;
Burgess et al., 1998; Chaytor et al., 2006; Wood & Liossi, 2006). These studies have found only
0-10% shared variance between any single EF test and an EF behavior rating scale (Barkley &
Murphy, 2010). This means that an EF test measuring a specific process in the lab (e.g., working
memory) may not correspond to daily functional impairments (which are measured by the
behavior rating scales). There is little consistency throughout the literature on the validity of the
tests of EF. Chaytor (2004) attempted to increase the ecological validity of EF tests. In this
study, and replicated in a later study (Chaytor et al., 2006), results indicated that when selecting
from the most reliable and valid tests of EF (WSCT, TMT, and Stroop), the most variance that
could be accounted for in impairments measured by behavior rating scales was 18-20%.
There are several areas of concern regarding the ability of EF tests to accurately identify
deficits in EF in daily life. As Barkley (2012) has described it, it is clinically useless to know
how a patient performs in the lab if that performance does not translate to the real-world and
their daily lives. Moreover, many of the definitions and theories of EF suggest an organization
of behavior, cross- temporally, towards a future goal (Fuster, 1997). It is unclear how EF tests
are able to determine progress towards a goal when testing sessions vary anywhere from five
minutes to a few hours, and most single subtests of a construct last less than an hour. This does
not mean that a test of EF does not measure problem solving over that course of time, but it may
35
do little to shed light on the daily difficulties of maintaining attention to a goal over days, weeks,
and years (Barkley & Murphy, 2010), such as would be needed in earning a college degree. An
additional problem with EF tests is the complexity in administering and interpreting the results
(Anderson, 2002; Castellanos, Sonug-Barke, Milham, & Tannock, 2006). Finally, as was
mentioned, the research is unclear as to whether intelligence significantly affects the results of
EF tests (Mahone, Pillion, Hoffman, Hiemenz, & Denckla, 2002). Although an association with
IQ would not preclude the use of EF tests, it would require that the norming of the EF measures
incorporate IQ as a factor, just as age is currently incorporated. This is not currently the practice
for test manufacturers.
EF Rating Scales of Impairment
Given the inherent problems with EF tests described above, other methods of measuring
EF have come to the forefront in the literature. EF behavior rating scales have grown in
popularity with clinicians, but not without continued controversy in the field. In general, a
behavior rating scale is a standardized set of items (questions) in which the informant makes
judgments about his or her behaviors or someone else’s behavior (Merrell, 1994). This relies on
direct observation of the person or self-perceptions of that specified behavior (Merrell, 1994). In
the case of an EF behavior rating scale, items focus on the different aspects of EF, including the
temporal and social components often lacking in EF tests. Behavior rating scales in general
assume that the informant has sufficient knowledge of the index person’s relevant behavior to
make the judgment (Fennerty, Lambert, & Majsterek, 2000) across considerably longer periods
of time (Barkley, 2011b). These items capture symptoms in a way that purports to improve the
ecologically validity as they happen in social contexts (Barkley, 2011b) rather than a snapshot in
time, such as an EF test. Barkley (2011b) also argued that EF behavior rating scales have more
36
face validity because the items are based directly from his theory of EF. As mentioned
previously, EF tests are likely contaminated by intelligence, and EF behavior rating scales are
not likely subject to this contamination. An additional motivation behind EF behavior rating
scales (and rating scales in general) is that they are generally more cost effective to administer
than EF tests (Barkley, 2011b). They do not require a specialized set of skills to administer.
Rating scales in general have the benefit of accessing an informant’s perspective with a vast
knowledge of the index person across time and when assessing infrequent behavior. Rating
scales can also tap a large variety of items and constructs within a relatively short time,
increasing their efficiency (Barkley, 2011b). With all the benefits, there are some disadvantages
to behavior rating scales in general. The informants themselves are also factors in the difficulties
experienced with behavior rating scales. The informant’s intelligence, insightfulness, education,
life experiences, and motivation all affect their ratings (Barkley & Murphy, 2010). Bias in the
informant’s intentions or malingering can affect rating scale outcomes, as can other psychiatric
symptoms. For EF behavior rating scales, symptoms of depression may be undistinguishable
from some ratings of EF deficit symptoms (Barkley, 2011b).
There are several EF behavior rating scales available and include the Behavior Rating
Inventory of Executive Functioning (BRIEF; Gioia, Isquith, Guy, Kenworthy, 2000), BRIEF-A
(adult version), the Comprehensive Executive Function Inventory (CEFI; Naglieri & Goldstein,
2013) and the Barkley Deficits in Executive Functioning Scale (BDEFS; Barkley, 2011b). The
BDEFS will be the focus of the remainder of this discussion. Because this study is a
psychometric evaluation of the BDEFS, extensive details are given regarding test development.
The BDEFS. The BDEFS is an 89 item behavior rating scale, utilizing a Likert scale (0-
rarely or not at all, 1=sometimes, 2=often, and 3=very often). The items are based largely on
37
Barkley’s theory of EF (Barkley 1997a, 2011c); however, understanding of these concepts from
other researchers (Denckla, 1996; Fuster, 1997; Welsh & Pennington, 1988) and an examination
of patients with injuries to their frontal lobes (Luria 1966; Burgess et al., 1998) were also a
foundation for items. There are five major constructs of EF measured by the BDEFS. These
constructs are noted in Barkley’s theory and in general, are listed in the literature as constructs of
EF (Castellanos et al., 2006; Willcutt, Doyle, Nigg, Faraone, & Pennington, 2005). The
constructs include the following: “inhibition; nonverbal working memory (self-directed sensing,
especially visual imagery, sense of time and time management); verbal working memory (self-
directed private speech, verbal contemplation of one’s behavior before acting, etc.); motivational
self-regulation (motivating one’s self during boring activities, etc.); and planning and problem-
solving (reconstitution, generativity, and goal-directed inventiveness)” (Barkley & Murphy,
2010, p. 41). There are two versions of the BDEFS: a self-report form and an other-informant
report form. The BDEFS has five factor scores: self-management to time, self-
organization/problem solving, self-restraint, self-motivation, and self-regulation of emotions.
Additionally, a total EF summary score and an ADHD-EF Index are given. The ADHD-EF
index is a separate score evaluating the likelihood that the individual may have adult ADHD
using 11 questions from the questionnaire (Barkley, 2011b).
Development of the BDEFS. The Barkley Deficits in Executive Functioning Scale
(BDEFS) was published in early 2011 after over a decade of development. The desire to have a
cost-effective means of conveniently capturing neuropsychological, behavioral, emotional, and
motivation symptoms often attributed to EF deficits was the motivation to develop this new
scale. Two federal grant-funded studies helped the development of the prototype of the BDEFS
(Barkley, 2011b). The first was the UMASS Study and examined clinic-referred adults with
38
ADHD, comparing these adults to a clinical control group and a community control group. The
clinical control group consisted of participants who were self-referred to the clinic to be
evaluated for ADHD, but who were not given a diagnosis based on subclinical symptomatology.
The second was the Milwaukee Study, which was a follow-up study of hyperactive children as
they entered young adulthood. Results of these two studies will be discussed in the following
sections. The original prototype consisted of 91 items developed to reflect inhibition, nonverbal
working memory; verbal working memory, emotional/motivational self-regulation, and
planning/problem solving. The focus of the BDEFS was to assess problem/deficit functioning
rather than normal functioning (Barkley, 2011b).
Barkley (2011b) published the final version of the BDEFS as an 89 item, paper-and-
pencil questionnaire, utilizing a Likert scale answer format (1-rarely or not at all, 2=sometimes,
3=often, and 4=very often). There are three separate instruments that were developed from the
original 91-item pool: self-rating scale (herein referred to as the BDEFS), other-informant rating
scale hereafter referred to as BDEFS-other, and a 20-item short-form. The self-rating scale and
other-informant scale are designed as described above. The short-form is a 20 item Short-Form
for screening purposes (Barkley, 2011b). The initial set of analyses discussed below were
conducted on the prototype BDEFS which has the original 91 items.
Initial factor structure validation. Several factor analyses were conducted (Barkley,
2011b) prior to the publication of the BDEFS. First, using the 351 clinic referred adults for an
ADHD evaluation from the UMASS study, a factor analysis revealed 10 factors with eigenvalues
over 1.00. However, only five of these factors had at least 10 items with high loading on a
factor. Three items did not have a loading ≥.400, so they were removed from the scale. The first
five factors accounted for more than 63% of the variance in the unrotated factor solution
39
(Barkley, 2011b). Also, one large factor accounted for the majority of EF deficits in daily life.
A varimax rotation was then conducted to see whether allowing the factors to correlate provided
a better fit. The factors from this initial data after rotation were:
Factor 1 (Self-management to Time) had 23 items and accounted for 15.7% of the
variance.
Factor 2 (Self-Organization/Problem Solving) had 21 items and accounted for 15.2%
of the variance.
Factor 3 (Self-Restraint or Inhibition) had 23 items an accounted for 14.1% of the
variance.
Factor 4 (Self-Motivation) had 11 items and accounted for 9.8% of the variance
Factor 5 (Self-Activation/Concentration) had 10 items and accounted for 8.6% of the
variance.
Even though these factors emerged, there were significant inter-correlations among
factors, ranging from .74 to .88 for the BDEFS and .75 to .88 for the BDEFS-other (Barkley,
2011b). Therefore, 56-77% of the variance was shared by the factors, which is in support of the
theory of an underlying metaconstruct of EF deficits (Barkley & Murphy, 2010). These above
two factor analyses are limited in that they do not represent the factor of Emotional Self-
Relegation, which was added later (Barkley, 2011b).
In analyzing the UMASS data, Barkley (2011b) determined that the adults with ADHD
rated themselves as having more severe EF deficits on all factors, compared to both the clinical
control group and the community control group. These results yielded statistically significant
differences on all of the factors. There appeared to be a progression of significant symptoms,
with the clinical control group endorsing more items, resulting in more significant deficits than
40
the community control group, in both the prototype of the BDEFS and the BDEFS-other. Using
the community control group, they were able to determine a cutoff for clinical norms for deficits
in EF, which was specified as 1.5 standard deviations above the mean of that group on the
prototype BDEFS (Barkley, 2011b). When using this cut-off, the group diagnosed with ADHD
had statistically significantly higher (more impaired) scores on the Self-Motivation sub-factor
than either of the other two groups. On all other factors, there was a statistical difference
between the community control group and the other two groups, but there was not a statistical
difference between the two clinic groups (clinical control and ADHD). This cut-off is useful to
differentiate between community controls and those with symptoms of ADHD (but not
necessarily diagnosed). Although this is not as useful as one might prefer, the prototype BDEFS
does differentiate a community sample, and the group with ADHD was higher than the clinical
control on all factors, just not statistically significant.
Up to this point, the prototype BDEFS had been used in all the analyses (Barkley,
2011b), which again lacked one of the five published factors (Emotional Self-Regulation). On
the BDEFS (published 89-item version), only items with a factor loading of at least .500 were
retained. Because of this trimming, the regulation of emotions scale was underrepresented, with
only a few items. The few items that were on the prototype BDEFS that were probing about
regulation of emotion loaded on the Self-Restraint/Inhibition factor. Therefore, additional items
were added using a model of self-regulation developed by Gross (1998). The version of the
scale on which the national norms are based had 100 items (with the above mentioned added in).
The final factor analysis, using the entire norming sample of 1249 adults (to be described
in a following section), was then conducted on the 100-items (Barkley, 2011b). The principal-
41
component factor analysis (PCFA), after rotation yielded the following factors with their percent
of the variance they accounted for:
Self-Organization to Time (Factor 1) - 13.9%
Self-Management to Time (Factor 2) - 12.0%
Self-Regulation of Emotion (Factor 3) - 10.2%
Self-Restraint (Factor 4) - 9.0%
Self-Motivation (Factor 5) - 8.1%
These five factors noted here are the final five factors in the published BDEFS that was
used in this study. However, of the 100 items used in the norming sample, only 89 were retained
on the final version of the BDEFS.
Norming procedures of the BDEFS. From the data collected from the normative
sample, the BDEFS factors were evaluated based on demographic characteristics of the sample.
Barkley (2011) enlisted a national survey company, Knowledge Networks®, to collect the survey
data used in the norming sample of the BDEFS. As noted above, the 100 item version was used
and then paired down based on the CFA. The sample was a Web-enabled Knowledge Panel®,
which is a method using a probability-based panel so that the sample is representative of the
population of the United States. A random sample of telephone numbers and residential
addressed was used. Then members of households were randomly sampled (Barkley, 2011b).
BDEFS-other surveys in the norming sample were not collected (Barkley, 2011b). An
exclusionary criterion for mental disorders was not employed. Therefore, results are based on a
true random sample of functioning in the adult population. A total of 1249 adults completed the
BDEFS. Participants spanned ages 19-81 and were equally distributed across the lifespan and
42
gender. Additionally, race/ethnicity, geographic location, education, income, marital status, and
employment status were all proportionate to the general population (Barkley, 2011b).
When looking at the relationship among the demographic factors and the BDEFS, gender
and age significantly correlated with some factors of the BDEFS. Age did not significantly
correlate with Self-Organization/Problem Solving; however, it did significantly correlate to a
small degree with the other factors and the total score. A possible reason for the differences
among age categories is that the prefrontal cortex is thought to not reach maximum potential
until the late twenties or early thirties (Barkley, 2011b). Additionally, there is thought to be a
decline in later adulthood, and again this is likely a reason the college student population may
show differences. When looking at the variable gender, women were significantly more likely to
show higher problems with Self-Regulation of Emotions. There was a marginal, and not a
statistically significant (p=.055) relationship between gender and Self-Restraint, with men
showing more problems than women. Gender was not significantly correlated with any other
factor or total score. Barkley (2011b) then looked to see if there was an interaction of gender by
age and there were no significant interactions.
Creation of the ADHD-EF Index. The ADHD-EF Index scale was developed given
previous research linking ADHD and EF deficits (Barkley, 2011b). The original data base of the
prototype BDEFS (UMASS study) was analyzed to see which items were most likely to identify
adult ADHD. A binary logistic regression of the group with ADHD and the community control
group yielded just five items needed to distinguish the groups with 98.1% accuracy (remember
from previously that the UMASS sample consisted of an ADHD group, a community group, and
a clinical control group). These five items correctly predicted that an individual in the ADHD
group was in fact ADHD 99.1% of the time and correctly predicted that an individual did not
43
have ADHD 96.9% of the time. The five items identified were: procrastinates or puts things off
until the last minute (item 1), trouble completing one activity before starting into another one
(item 16), trouble organizing my thoughts (item 24), difficulty changing behavior when I am
given feedback about my mistakes (item 50), and take short-cuts in my work and do not do all
that I am supposed to do (item 65). The same analysis was then used to distinguish between the
ADHD group and the clinical control group. Again, the clinical control group consisted of a
group of participants who were referred for an ADHD evaluation (believed they had ADHD), but
did not receive a diagnosis (based on the DSM-IV; American Psychiatric Association, 2000) of
ADHD. In this analysis, seven items were needed to best differentiate the groups, with an
accuracy rate of 72.3%. These seven items correctly identified an individual as ADHD 56% of
the time and correctly identified the individual in the clinical control group 82.9% of the time.
These seven items were: trouble planning ahead or preparing for upcoming events (item 6),
trouble motivating myself to work (item 13), difficulty stopping my activities or behaviors when
I am given feedback about my mistakes (item 49), difficulty changing my behaviors when I am
given feedback about my mistakes (item 50), not aware of things I say or do (item 55), more
likely to drive a motor vehicle much faster than others (item 60), and depends on other to get my
work done (item 69). This indicated that it was more difficult to distinguish the clinical control
group from the ADHD group with rating scales alone (accuracy of 72.3%) than it was to
distinguish the ADHD group from the community control group (accuracy of 98.1%). Of the
seven items needed to identify between the ADHD group and the clinical control group, only one
was a duplicate (difficulty changing my behavior once given feedback, item 50) of the five items
needed to distinguish between the ADHD group and the community control group. The five
items found to distinguish the ADHD group from the clinical community controls was added to
44
the seven items used to distinguish the ADHD group from the clinical controls (less one
overlapping item) for a total of 11 items. These items constituted the ADHD-EF Index (Barkley,
2011b). The 11-item ADHD-EF index has not been replicated on multiple age categories or
severity levels of ADHD, or in the college student population. However, the 11-item ADHD-EF
index was subsequently tested on the norming sample (see previous section for description of the
normative sample). The participants in the norming sample were given an ADHD symptom
checklist along with the BDEFS. The participants in the top fifth percentile of ADHD symptoms
were classified as the ADHD group. The ADHD-EF Index was accurate in predicting this group
94% of the time (Barkley, 2011b).
Reliability
Inter-observer agreement and disparity. In the UMASS study, self-report and other-
informant report data were collected (Barkley, 2011b). The correlations between self- and other-
informant reports had a reasonable level of agreement with r between .66 and .79 across the four
factors (this was the prototype BDEFS without the Self-Regulation of Emotions scale). Other-
informant reports were not collected in the normative data; therefore, the agreement between the
two remains to be evaluated with the final five-factor version of the BDEFS. In the UMASS
study, the community control group (the group most similar to the norming sample) had
relatively little discrepancy between the self-report and the other-report. However, there was a
larger than expected standard deviation around the mean. For the clinical control and ADHD
groups, the disparities between the BDEFS and BDEFS-other were significantly higher, but they
did not differ between the two groups (ADHD versus clinical control). The absolute values of
the disparities between the BDEFS-self and BDEFS-other were taken in this case, so the
45
direction of the disparity was not taken into account. Additionally, age and IQ did not affect the
disparity between the BDEFS-self and BDEFS-other (Barkley, 2011b).
Internal consistency and test-retest. A Cronbach’s alpha was conducted to analyze
internal consistency of the BDEFS and it was found to be .918 for the Total EF Symptoms. The
Cronbach’s alpha for the five factors ranged from .914 to .958. The ADHD-EF Index was .842.
A test/retest comparison was completed on 62 randomly selected participants and was adequate
at .62 to .90 for the five factors and .70 for the Total EF Symptoms Score. Both of these
analyses were conducted on the full BDEFS from the norming sample (Barkley, 2011b).
Validity
Discriminant Validity. Data from the UMASS (prototype BDEFS) study showed that
80-98% of adults with ADHD were in the clinical range (above the 93rd percentile, 1.5 SD above
the mean) across the various factors of the BDEFS, versus only 8-11% across the various factors
in the community control group (Barkley, 2011b), using the self-report data. At first glance, this
appears to have excellent discriminant validity in distinguishing between adults with ADHD and
the control group. However, the clinical control group (those who believed they had ADHD, but
were not diagnosed), were in the clinically significant range at a rate of 83-98%. This could be
the result of the self-report aspect of this measure. At least 45% of the adults in the clinical
control group endorsed enough symptoms of ADHD to meet diagnostic criteria, if based solely
on self-report information. Therefore, if they are likely to endorse a high number of symptoms
on the ADHD criteria scale, they are equally as likely to do so on the BDEFS. In addition,
people who are willing to seek help (e. g., those who self-refer to a clinic for evaluation) could
present differently than those who do not seek help for a variety of reasons. The issue of
secondary gain or malingering could be at play here in over-endorsing symptoms. This analysis
46
should be replicated on a population who is already diagnosed with ADHD and does not have the
potential of a secondary gain. Additionally, looking at a clinical population who does not
believe they have ADHD may also be beneficial. The evidence given above does support that
the BDEFS is adequate at discriminating the normal population from the clinical population
(Barkley, 2011b); however, further evaluation is needed to distinguish between the clinical
populations. Additionally, these were all conducted with the prototype BDEFS, rather than the
final five-factor BDEFS.
ADHD-EF Index Validity. Again, in looking at results from the UMASS study, the
ADHD-EF Index was a good predictor of adults with ADHD. In fact, 98.5% of the group with
ADHD had a score above the 93rd percentile cutoff on the ADHD-EF index. However, the
clinical control group was also high with 96.6% above the same cutoff. Only 7.6% of the
community control group was above the 93rd percentile Barkley, 2011b). Given that the ADHD-
EF index was created based on its ability to discriminate those with an ADHD diagnosis, it is
expected that it would show good predictability (Barkley, 2011b).
At this time, no studies have been conducted using the BDEFS to discriminate between
control groups and other clinical groups that may have EF deficits, such as neurological disorder
and traumatic brain injuries, and there is a need for further research to be conducted.
Additionally, there have been no studies on the college student population, and as reported, the
UMASS study used the prototype BDEFS, rather than the finalized BDEFS.
Criterion Validity. Severity of ADHD symptoms is one of the most researched areas
with the BDEFS (Barkley, 2011b). As was mentioned, the correlation was significant for the
Total Score on the BDEFS with ADHD symptom criteria. Total ADHD symptoms were
measured by self-report on the Barkley Adult Rating Scales for ADHD, which is based on DSM
47
criteria (Barkley, 2011b). The ADHD symptoms in the above mentioned analysis were the total
symptoms (combining inattentive, hyperactive, and impulsive symptoms); however, further
analysis was conducted on inattentive symptoms and hyperactive/impulsive symptoms
separately. When looking at the symptoms for each subtype separately, there were also high
correlations with the BDEFS items. For inattentive symptoms of ADHD, the correlations ranged
from .80 to .92 across the five factors and the ADHD-EF Index of the BDEFS. For
hyperactive/impulsive symptoms of ADHD, the correlations were slightly less at .68 to .71
(Barkley, 2011b) across the five factors and the ADHD-EF Index. These statistics were analyzed
using both the UMASS sample and the Milwaukee sample, but they have not been replicated by
someone other than the author of the rating scale.
Construct Validity. A factor analysis was conducted using the UMASS (prototype
BDEFS) sample to see if ADHD symptoms and BDEFS symptoms were measuring the same
construct (Barkley, 2011b). As was discussed in the literature review of EF, Barkley’s Extended
Phenotype Theory proposed that ADHD and EF deficiencies were the same thing (Barkley,
2012). If this is the case, the factor analysis should show one construct when using ADHD
symptoms and the BDEFS items. The analysis showed a high factor loading onto one construct,
adding support for Barkley’s theory that ADHD and EF were just different names for the same
construct. The author then replicated this factor analysis with the normative sample (the
participants also completed the Barkley Adult Rating Scales for ADHD) and the analysis was
identical (Barkley, 2011b). These analyses should be replicated by a researcher other than the
author to continue to show support for this theory.
Limitations of Rating Scales. To summarize, there are a plethora of tools, tests, and
rating scales that purport to measure EF. Many of these EF tests and behavior rating scales have
48
evidence supporting validity and reliability of their results. The multiple and opposing models of
EF have also led to significantly different approaches to measuring EF. This section reviewed
common EF tests and behavioral rating scales commonly used to assess EF, along with the
benefits and consequences associated with each type. Additional research into the current
measures that are available with tighter control of the methodologies is needed. Finally,
behavior rating scales, specifically the BDEFS, are relatively new in the field and require
substantial research to collect evidence of validity and reliability. In summary, it appears that the
administration of both EF tests and behavior rating scales may add incrementally to our
understanding of the EF of an individual. The EF tests are less influenced by the patient’s level
of insight or potential for secondary gain, yet they are not tied to a specific diagnosis. Behavior
ratings scales, on the other hand, have greater diagnostic sensitivity in higher functioning
individuals. However, they may be vulnerable due to their higher face validity (allowing for
intended or unintended manipulation) and the necessity for patient insight.
Of the available behavior rating scales to measure EF, the BDEFS has recently been
published. To date, there have been no independent research studies published to provide
additional evidence of validity and reliability. To add to the body of research on the BDEFS,
there is a need to evaluate the types of test constriction and validation principles which are
required to provide the psychometric properties of a new test.
Test Construction and Validation Principles
This section will describe important considerations in attempting a psychometric
evaluation of a test, specifically as it relates to the BDEFS.
49
Reliability
The reliability of a measure refers to the degree to which the score produced by the
measure is reproducible (Reis and Judd, 2000). There are four classical primary types of
reliability: inter-rater reliability, split-half reliability, internal consistency, (Carmines & Zeller,
1979; Heiman, 2002), and alternate format (Carmibes & Zeller, 1979). Reliability is important
in multiple ways when constructing and utilizing measurements. Given self-reporting of
executive functioning symptoms, inter-rater reliability is an important factor in determining the
reliability of the measure. As has been suggested, individuals with deficits in EF may lack self-
awareness about their symptoms. Since the BDEFS is a self-report measure of these symptoms,
inter-rater reliability gives the interpreter a way to compare self-reported symptoms to other-
informant reported symptoms. When a measurement tool is reliable, it generally means that the
error of the measurements is reduced (Goodwin, 2010). Additionally, reliability is correlated
with improved validity (Fink & Litwin, 1995). After determining support for reliability of a
measure, determining validity becomes a priority.
Validity
Once a measure shows evidence of reliability, it requires support for validity. Does the
measurement tool actually measure what it is intended or purported to measure? To quote
Messick (1989), the question of validity can be summarized as “to what degree-if at all- on the
basis of evidence and rationales, should the test scores be interpreted and used in the manner
proposed?” (p. 5). Given the importance placed on results of measurement tools in academia and
in clinical practice, using a measure that has support of validity is critical. Broadly speaking,
validity speaks to the empirical and theoretical basis for the interpretation of test scores
(Messick, 1989, p. 6). Beyond this, Messick suggests that all methods of validation are
50
variations on the scientific method. This extends to inferences (comparable to hypotheses) and
validation (comparable to hypothesis testing). These inferences regarding the scores are only
meaningful in the context of theory, which is again consistent with the way in which information
is interpreted and conclusions are drawn within the scientific method (Messick, 1989). Once
there is evidence to support the assertion that the measure appears to be assessing what it is
intended to, inferences can be made regarding its use (Fink & Litwin, 1995). There are many
different types of validity to add support to the idea of whether a measure is actually measuring
what it is purported to measure. The four primary types of basic validity are face validity,
criterion validity, content validity, and construct validity.
Face validity merely means that the test appears to be valid based on face-value (Fink &
Litwin, 1995; Heiman, 2002). Criterion validity is the degree to which a measure correlates with
a behavior that the individual is either currently presenting (concurrent) or presents in the future
(predictive) (Jackson, 2008; Heiman, 2002). This usually includes an empirical correlation
between an observable (or measureable) behavior and the measurement tool being validated
(Messick, 1989). Content validity refers to the extent to which the items produce responses that
measure or represent that of the construct or concept (Akien, 2003). This is usually
accomplished by an expert judgment based on opinion, rather than empirical evidence. None of
the above types of validity are concerned with differences in responding across setting or groups,
internal or external structure, or the social consequences of using such a measure (Messick,
1989).
Construct validity, however, is evidence to support that the measure tests the actual
construct that it intends to measure. This is important given that clinical impressions (social
consequence as Messick puts it) are often made from measures, such as the BDEFS. Within
51
construct validity, there are two types: convergent and discriminant validity. Convergent
validity refers to the correlation of the scores in question to another procedure which is already
accepted as valid measuring the same construct. Discriminant validity refers to extent to which
the scores obtained are not correlated with those of an accepted procedure that measures other
variables (Heiman, 2002). In construct validity, the score resulting from the measure is one
piece of the many different indicators that represent the construct (Messick, 1989). It is
important to remember when conceptualizing construct validity that the relationships between
criterion and content validity as well as the empirical evidence from construct validity should
have a theoretical commonality (Gulliksen, 1950). These things together, combining the
scientific underpinnings and the ethical nature of social consequences, form the basis for modern
test theory (AERA, APA, & NCME, 1999; Messick, 1989).
Construct validity has two main threats: construct underrepresentation and construct-
irrelevant variance. If the items on the measurement tool are too narrow and do not cover the
depth of the construct, underrepresentation occurs. On the other hand, if items are harder or
easier for one group than another on a variable that has nothing to do with the construct, then
irrelevant variance is introduced (Messick, 1989). When a new measure is published (such as the
BDEFS) it is important that before serious clinical conclusions are drawn, evidence of validity is
provided (Fink & Litwin, 1995). This includes evidence to support construct validity in multiple
populations (Messick, 1989; Heiman, 2002; Fink & Litwin, 1995), as well as from independent
researchers.
Discriminant Function Analysis
One way to determine whether a measure discriminates between two or more groups is a
discriminant function analysis. This is used when the goal is to analyze a relationship between a
52
dichotomous dependent variable and either a continuous or dichotomous independent variable.
The discriminant analysis endeavors to use the independent variable(s) to discriminate or
distinguish among the dependent variable groups (Myers, Gamst, & Guarino, 2006). In this
case, ADHD or non-ADHD. Another analysis that is similar to the discriminant function
analysis is logistic regression. Both of these techniques predict group membership; however, in
a logistic regression, the independent variable can be either continuous or dichotomous.
Additionally, a logistic regression is non-linear, and a discriminant function analysis is linear
(Myers et al., 2006).
An important feature of a discriminant function analysis is that it yields an accuracy rate,
which indicates how useful the tool is in determining group membership. In doing so, this
analysis helps to establish the boundary of the groups (Myers et al., 2006). To accomplish the
accuracy rate, eigenvalues are used, as they are in confirmatory factor analysis (Myers et al.,
2006).
Confirmatory Factor Analysis
In attempting to provide evidence of construct validity of a new measure, factor analysis
is a popular statistical procedure. It provides support for the factor structure of a measure when
attempting to provide valid information on latent factors. A latent factor is a construct that
describes a set of symptoms or behaviors measured by the measurement tool (e. g. self-
motivation to time) (Brown & Cutik, 1993; Brown, 2006). There are two basic types of factor
analyses: exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).
Traditionally, an EFA is used in the preliminary analysis of a new measure. It is exploratory in
that it uses a data driven approach to determine the number of factors and the nature of those
factors (Brown, 2006). Once empirical and theoretical frameworks have been established, a
53
CFA is then the statistic of choice. A CFA uses a priori decisions based on the empirical
evidence (from an EFA) and theory to determine the number and nature of the parameters. The
CFA is an essential statistical procedure for providing evidence for construct validity. It
produces persuasive evidence for convergent and discriminant validity of the theoretical
concepts of the latent factor or construct (Brown, 2006).
There are other statistical procedures to examine the relationship among variables and
factors, such as multiple regression and correlations. However, the CFA has an advantage over
these procedures as it is the only one to take into account error variance. The CFA’s primary
intent is to determine the nature and number of latent variables, or factors (Floyd & Widaman,
1995). CFA is a structure equation model (SEM) which deals specifically with measurement
models and the relationships between observed measures or indicators and the latent variables
they propose to represent (Brown 2006). There are multiple ways to extract the factors when
conducting a CFA. These include “maximum likelihood, principal factors, , imaging analysis,
unweighted least squares, generalized least squares, minimum residual analysis, weighted least
squares, and alpha factor” (Brown, 2006. P. 21), to name a few. The maximum likelihood is the
most common because it has the ability to reproduce the factors and provides a goodness of fit
model (Brown, 2006).
Proposed Study and Research Questions
As was laid out in the literature review, college students with ADHD differ from their
adult counterparts in both the demands placed on their daily activities (more heavily EF based in
general), and their different characterological attributes. While determining whether a college
student has a diagnosis of ADHD is complex, it is also an important factor given the rate of
academic difficulties and drop-out in this population. In addition, the cost of misdiagnosis is
54
great. Deficits in EF have been related to the diagnosis of ADHD in the literature, both on the
biological level and theoretical level. fMRI studies have shown the relationship between ADHD
and EF deficits time and again. There are a range of EF tests that purport to measure EF;
however, this method of measurement is flawed at best. Given this lack of appropriate
measurement, behavior rating scales have come to the forefront in the quest in assessing EF. The
BDEFS is a new EF behavior rating scale which has been published recently. It has yet to have
evidence of validity and reliability through independent sources, and has not been researched in
the college student population.
The current research study proposed to add to the support of construct validity, reliability,
and diagnostic usefulness of the BDEFS. The BDEFS purports to measure the latent traits of
executive functioning: self-management to time, self-organization/problem solving, self-
restraint, self-motivation, and self-regulation. The BDEFS also purports to have diagnostic
validity in determining the likelihood that an adult individual has ADHD. Given that this is a
new measure and little has been done in the way of support of validation, this study set out to
contribute to the literature regarding the usefulness of this measure for the college student
population. The research questions that were proposed are as follows:
1. What is the relationship between the BDEFS self-report and other-informant report in a
college student population of students with ADHD? What are these relationships on the
following factors: Self-Management to Time, Self-Organization and Problem Solving,
Self-Restraint, Self-Motivation, Self-Regulation, ADHD-EF Index, and Total Executive
Functioning Symptoms? How do these correlations compare to the correlations that
Barkley found in his original study? Are the means of the self-informant reports higher or
55
lower than the means of the other-informant reports within the college student population
of students with ADHD?
2. In the college student ADHD sample, is there a correlation between intellectual functioning
(as measured by the BIA) and the BDEFS similar to the correlation between the intellectual
functioning and BDEFS in the UMASS study. This relationship was analyzed for the
following factors: Self-Management to Time, Self-Organization and Problem Solving, Self-
Restraint, Self-Motivation, Self-Regulation, and Total Executive Functioning Symptoms.
3. Are the same ADHD-EF Index items the most predictive of a diagnosis of ADHD in a
college student population as they are in the original normative sample? The current
BDEFS ADHD-EF Index is composed of 11 items. This index was created using a
discriminant analysis to select those items (out of 89) that best discriminate those with
ADHD from a normative sample. Do the same 11 items best discriminate those with
ADHD from a normative sample in a college student population?
4. Is the factor structure of the BDEFS the same in college students as it is for the normative
sample, based on a confirmatory factor analysis?
56
CHAPTER 3
METHODS
Introduction
In the literature review, the difficulties in diagnosing college students with ADHD, the
difficulties with deficits in EF, and the importance of test validation were laid out. This chapter
describes the Barkley Deficits in Executive Functioning Scale (BDEFS) and the development
process of this scale in more detail. Additionally, the methods used to recruit participants and
the analyses used are discussed below.
Participants
The 596 participants were all college students. These students were either recruited to
participate in the control group (n=459) or were college students ADHD (n=137). The
participants attended a large, highly selective southeastern university in the United States. The
demographic breakdown of the groups with ADHD and without ADHD is located in Table 1.
The gender breakdown in this sample was consistent with the published statistics of the
university where the data were collected. All students with ADHD in this sample were either
inattentive type or combined type ADHD. There were no participants in this sample with
hyperactive/impulsive type. The absence of students with the hyperactive/impulsive subtype is
consistent with the literature on adults and college students with ADHD (Heiligenstein et al.,
1999).
Participants in the ADHD group were part of an archival data set, who were recruited
through a campus clinic where they had been referred for an evaluation to determine whether they
had a diagnosis of ADHD. These participants were clients who, through the course of their
evaluation, completed the BDEFS survey while they were students. The students who were given a
57
diagnosis of ADHD were coded in the ADHD group. Participants who had co-occurring disorders
are included in the ADHD group as ADHD is highly comorbid with many other DSM disorders
(APA, 2013). The students who did not receive a diagnosis were excluded from the sample for
most analyses, with the exception of the confirmatory factor analysis. For the ADHD group, only
students who had previously given consent to participate in research were utilized (HSC approval
2012.7742). Please see appendix A for informed consent and for IRB approval.
Table 1 Demographics
Demographic Descriptors
Total Sample n=596
M (SD)
ADHD Group n=137
M (SD)
Non-ADHD Group n=459
M (SD) Age 20.71 (2.46) 21.5 (3.12) 20.53 (2.26)
Female 63.4 52.8 65.8
Male 36.6 47.2 34.2
Freshman 17.6 19.4 17.2
Sophomore 28 29.6 27.7
Junior 29 27.8 30.3
Senior 24.5 23.1 24.8
Caucasian 66.6 63.7 67.1
African American 16.5 14.7 16.8
Asian 3.6 2 3.9
Hispanic 10.4 15.7 9.2
Other 3.1 3.9 2.9
Note: the numbers represented are percentages for each sample (gender, year in college, and
ethnicity). The numbers for the variable age, are represented in years
Diagnosis of ADHD
As part of the diagnosis of ADHD in the preceding archival data set, the clients completed the
following forms created by Barkley (2011a): Employment History, Developmental History, Social
58
History, Work Performance Rating Scale, Health History, Current (ADHD) Symptoms Scale (self-
report form and other-informant form), Childhood Symptoms Scale (self-report form and other-
informant form), Driving Behavior Survey, and the Driving History Survey. Additionally, the clients
completed a pencil and paper copy of the BDEFS (Barkley, 2011b). The clients also completed the
Academic Success Inventory for College Student (ASICS; Prevatt, Wells, Festa-Dreher, Yelland, &
Lee, 2011) and a checklist of symptoms based on the DSM to rule-out other mental health diagnoses.
Additionally, other-informant reports were collected using the BDEFS-other. Clients were instructed
to select someone to be their other-informant who knew them well and interacted with them on a
regular basis. Additionally, the other-informant should have known the client for a minimum of six
months. Generally speaking, the other-informant was a significant other, best-friend, roommate,
sibling, or parent. In most cases, the other-informant report form was emailed directly to the
informant through an online survey. The results were then sent directly back to the clinic through the
survey management software. This was done to improve the likelihood that the informant would be
forthcoming and not concerned about the participant’s feelings when rating the impairment. A
clinical interview was conducted with the client to provide personal anecdotes corroborating their self-
reported symptoms on the rating scales. As far as psychometric testing was concerned, the client was
administered three subtests from the Woodcock-Johnson III Tests of Cognitive Abilities (WJ-III
COG; Woodcock, McGrew, & Mather, 2001b). This provided an estimate of the client’s cognitive
processing abilities. The subtests were: Verbal Comprehension, Concept Formation, and Visual
Matching. These three subtests taken together form the Brief Intellectual Ability (BIA) cluster.
According to Schrank, Mather and Woodcock (2011), the reliability coefficients for the BIA
ranged from .94 to .98. Concurrent validity with other measures of intelligence (full IQ
measurements) was reported to range from .60-.69 (McGrew & Woodcock, 2011), which falls in
59
the moderate to acceptable range (Heiman, 2002). In addition, three subtests from the Woodcock-
Johnson III Tests of Achievement (WJ-III ACH; Woodcock, McGrew, & Mather, 2011a) were
administered to evaluate academic achievement. These subtests included Understanding Directions,
Passage Comprehension, and Reading Fluency. These brief measures of cognitive abilities and
academic achievement has an age range from 2 to 90+ years old. The achievement testing data
was not utilized in this study.
A diagnosis of ADHD was made if the client met the criteria set out by Barkley (2011a) and
the DSM-IV-TR: (a) there was evidence that the client experienced symptoms of ADHD in early
childhood, (b), these symptoms appeared no later than middle school and impaired their
functioning across multiple settings, (c) there was evidence that the client was currently
experiencing symptoms of ADHD, which cause marked and chronic impairment across settings,
and (d) there were no explanations other than ADHD that better accounted for the client’s current
symptoms. The process of determining whether a student met the above ADHD criteria was a
clinical process which took into account not only checklists, but a lengthy diagnostic interview
gathering information from multiple aspects of the student’s life. No client was included into this
data set where the diagnosis was made with the newest addition of the DSM (DSM-5). However,
there were approximately 20 students who did not receive a diagnosis of ADHD that were collected
during this time frame of this data collection, and they are not included in this sample. Some of
these students may have been eligible for a diagnosis of ADHD under the new criteria in the DSM-
5.
The evaluators were graduate-level students working towards a master’s degree in School
Psychology or a doctoral degree in Combined Counseling Psychology and School Psychology. All
cases were supervised by both an EdS level School Psychologist and a doctoral level Clinical
60
Psychologist. Agreement between the evaluator and the two supervisors was required for a
diagnosis of ADHD.
The control group was a combination of archival data and newly collected data from the
same university as the group with ADHD. The Educational Psychology and Learning Systems
Department at the university maintains a subject pool to aid in recruiting participants. However,
this subject pool was mostly female; therefore, additional recruitment procedures were taken to
increase the size of the sample and to balance the gender disparity. This was done by recruiting
from other places on campus such as the library, student union, and club-sports. In addition, several
classes were selected based on instructor approval to be offered the opportunity to participate.
Classes for recruitment included: General Biological Sciences for non-majors (BSC 1005),
Principles of Macroeconomics (ECO 2012), Principles of Microeconomics (ECO 2013), Race and
Ethnicity in the US (AHM 2097), and Dynamic Earth (GLY 1000). Completion of the BDEFS
survey (Appendix B) was voluntary for all students and the informed consent can be found in
Appendix C. Human Subjects Approval (HSC No. 2013.10087) is in Appendix D. The survey
also requested demographic information from the participant such as gender, ethnicity, and year in
college (see appendix E).
For the participants in the control group, the data was collected via an online survey or by a
paper and pencil copy. The BDEFS and the demographic questions were converted from the
original paper-format to an online-format to make the completion easier for students; however, a
paper copy was also available for participant choice. Participants were either provided extra credit
in their classes, fulfilled a class research requirement, or entered into a lottery for a $15 gift
certificate to the store of choice. Additionally, participants had the option to choose a gift card for a
free coffee drink from the on-campus coffee shop for completing the survey. The type of incentive
61
depended on the instructor preference for their students and the setting in which the recruitment
occurred. The participants provided their names and contact information in a separate survey that
was hyperlinked to the original survey if they selected the lottery option. This ensured
confidentiality for participant names and contact information. The personal information was kept in
a separate data file and could not be matched up with their responses to the original survey after the
initial search of duplicate names.
The survey tool prompted students to answer each question; therefore, in the control
sample, there was no missing data. In the archival data set, the evaluator was to check all forms to
ensure they were complete, and there were very few pieces of missing data. The missing data
points were replaced with the mean score for that variable.
All participants were enrolled in classes, at least part-time, and students under 18 years of
age and older than 30 years of age were excluded. These cut-offs were selected given the purported
changes in the development of the EF system at around age 30 (Barkley, 2012). Graduate students
were also excluded from this study given that their age range is generally older, and their general
characteristics may be fundamentally different from that of an undergraduate student. A search of
participant names was conducted to make sure there were no duplicates participants (a student who
visited the clinic and was in one of the control group collection areas).
Hypotheses and Planned Data Analyses
Preliminary Analysis
Archival data set- (ADHD group). The archival data set was coded into the Statistical
Package for the Social Sciences (SPSS), version 18 for Windows. The appropriate variables
were extracted from that data set and merged with the control participant data. A search for data
points that fell outside the appropriate range for that particular variable was conducted, as well as
62
a search for missing data. All missing or inaccurate data was corrected by pulling the participant
file and recoding the data if available. A listwise deletion process was used to handle missing
variables from a participant. This procedure is appropriate given that there was small amount of
missing data (Graham, 2009) because the data was being collected by the evaluators during the
evaluation session.
Given that a percentage of the college student population has ADHD, approximately five
percent of students recruited for the non-ADHD group reported a prior diagnosis of ADHD
(based on results from the demographic section). Students who reported a prior diagnosis were
only used in the confirmatory factor analysis and were not used in any other analyses.
Newly collected data (non-ADHD group). The BDEFS and demographic questions had
previously been collected from a sample of college students. This data was collected through an
on-line survey management tool and then was converted into an SPSS file though the survey
management system. For the surveys collected with paper and pencil, this researcher coded the
data into Excel. These data sets were then merged with the archival data set and again, a search
for missing and inaccurate data was conducted. The data set did not have out-of range data or
missing data given that the survey tool was set to disallow students to skip questions and the
answers were in multiple choice format rather than open-answer.
Planned Analyses
Research Question 1. What is the relationship between the BDEFS self-report and
other-informant report in a college student population of students with ADHD? What are these
relationships on the following factors: Self-Management to Time, Self-Organization and
Problem Solving, Self-Restraint, Self-Motivation, Self-Regulation, ADHD-EF Index, and Total
Executive Functioning Symptoms? [Both the original-11-item ADHD-EF-index and the new 15-
63
item-ADHD-EF index (see research question three below) were included in this analysis.] How
do these correlations compare to the correlations that Barkley found in his original study? Are
the means of the self-informant reports higher or lower than the means of the other-informant
reports within the college student population of students with ADHD?
In research question one, only the ADHD group was utilized. That group had 137
participants who had given informed consent to be included in this research study. To answer
the first part of this question, a Pearson product-moment correlation was utilized to determine the
relationship between the self-informant report (BDEFS-self) and the other-informant form
(BDEFS-other) of the five factors, total score, and both ADHD-EF Indexes. The hypothesis for
this part of the analysis was that the college student sample would yield adequate inter-rater
agreement, based on the fact that Barkley obtained significant correlations when he looked at
inter-rater agreement in his original analysis. In addition to the simple correlation, a Fisher r-to-z
transformation was conducted using Barkley’s (2011b) reported correlations and the correlations
found in this study to see if there was a statistically significant difference between the two
(Lowry, R., 2013). Finally, t-tests were conducted to see if the means for each factor separately
were statistically significantly different between the BDEFS-self and the BDEFS-other. An a
priori power analysis was conducted to determine a suitable sample size for this statistical test.
The G-Power 3.1.7 program (Faul, Erdfekder, Buchner and Lang, 2013) was utilized to conduct
the power analysis using two-independent pearson correlations, with a projected large effect size
of 0.66 and an alpha error probability of 0.01, and a power value of 0.8, An alpha level of .01
(.05 divided by 5 tests = .01) was used due to the need for a Bonferonni correction to correct for
family-wise error rate. There are five factors (the ADHD-EF Index and Total Score are
combinations of the same items). Barkley showed a large effect size between the self-form and
64
other-informant form when he ran the correlation on his norming sample (.66-.79); therefore, it is
presumed that a large effect size will be found in this analysis as well. The power analysis
indicated that 120 participants were needed. For the t-tests that were conducted to see if there
was a statistically significant difference in the means for each factor between the BDEFS-self
and the BDEFS-other, the p-value will be .01 rather than .05 as well given the same need as
above to use a Bonferroni correction.
Research Question 2. In the college student ADHD sample, is there a correlation
between intellectual functioning (as measured by the BIA) and the BDEFS similar to the
correlation between the intellectual functioning and BDEFS in the norming sample? This
relationship was analyzed for the following factors: Self-Management to Time, Self-Organization
and Problem Solving, Self-Restraint, Self-Motivation, Self-Regulation, and Total Executive
Functioning Symptoms.
To answer this question, a Pearson product-moment correlation was utilized to determine
the relationship between the BIA and the five factors and total score of the BDEFS. As with the
previous research question, only the ADHD group was utilized in this analysis.
The same a priori power analysis was conducted to determine a suitable sample size for
this statistical test as the previous research question.
Research Question 3. Are the same ADHD-EF Index questions the most predictive of a
diagnosis of ADHD in a college student population as they are in the original normative
sample? The current BDEFS ADHD-EF Index is composed of 11 items. This index was created
using a discriminant analysis to select those items (out of 89) that best discriminate those with
ADHD from a normative sample. Do the same 11 items best discriminate those with ADHD
from a normative sample in a college student population?
65
As discussed previously, the original 11-item ADHD-EF Index was developed by using
two separate analyses comparing different samples. One analysis compared an adult sample of
participants with ADHD to a clinic referred sample who did not qualify for a diagnosis of
ADHD. The items that were yielded from that analysis are labeled below as (CL). The next
analysis determined items that best discriminated the group of adults with ADHD from a
community sample. The items yielded in this analysis are labeled below as (COM). The original
11-item ADHD-EF Index is composed of the following items:
Procrastinate or put off doing things until the last minute (item 1; COM)
Have trouble planning ahead or preparing for upcoming events (item 6; CL)
Have difficulty motivating myself to stick with my work and get it done (item 14; CL)
Have trouble completing one activity before starting into a new one (item 16; COM)
I have trouble organizing my thoughts (item 24; COM)
Have difficulty stopping my activities or behavior when I should do so (item 49; CL)
Have difficulty changing my behavior when I am given feedback about my mistakes (item
50; COM; CL)
Not aware of things I say or do (item 55; CL)
More likely to drive a motor vehicle much faster than others (excessive speeding) (item 60;
CL)
Likely to take short cuts in my work and not do all that I am supposed to do (item 65; COM)
Have to depend on others to help me get my work done (item 69; CL)
In Barkley’s normative sample, the 11 items above best discriminated those adults who
had symptoms of ADHD from clinic-referred adults without ADHD and from a community
66
sample without ADHD. The question at hand is whether, in a college sample, the same 11 items
best discriminate those with ADHD from those without ADHD.
The Barkley analysis comparing the ADHD group with the community control group
(COM) is the most closely matched to the sample in this study; therefore, these items (1, 16, 24,
50, and 65) were analyzed separately to see how well they match the analysis conducted by
Barkley and will be referred to as the 5-item community control ADHD-EF Index. In summary,
this analysis was conducted twice, once using all 11 items and once using only the 5 items from
the community control norming group.
The current analyses used discriminant function analysis. Logistic regression and
discriminant function analysis are very similar in that they predict the likelihood of a
dichotomous group membership (ADHD group or Control group). In a logistic regression, the
predictor variables (the items in this case) are either continuous or dichotomous; however, in a
discriminant function analysis, they are always continuous (Meyers, Gamst, & Guarino, 2006).
Additionally, if the model is non-linear, then a logistic regression should be used. However,
results indicated that the model was in fact linear. Therefore, the more appropriate statistical
analysis is the discriminant function analysis (Meyers, Gamst, & Guarino, 2006).
The discriminant function analysis is a type of general linear analysis; therefore, similar
assumptions must be met. These include normality, linearity, non-multicollinearity, independent
predictors, and homoscedasticity (Meyers, Gamst, & Guarino, 2006). While the assumptions are
relevant for a DFA, the DFA makes fewer statistical demands than does the MANOVA. In the
case of a MANVOA, inferences are being made. However, if one achieves high classification
rates in a DFA, the shape of the distributions is less important (Tabachnick and Fidell (2007).
67
In addition to group identification, sensitivity and specificity were analyzed through the
discriminant function analysis. Sensitivity denotes how well the identifiers accurately detect
whether a participant belongs in a group. In contrast, specificity indicates the accuracy of the
cutoff score in excluding participants who do not belong in the group (Hood & Johnson, 2006).
This is expressed as a percentage of accuracy for each.
There is discrepancy in the literature about the appropriate sample size for a discriminant
function analysis. According to Meyers, Gamst, and Guarino (2006) the minimum requirement
to run this statistical procedure is: the maximum number of independent variables = N-2, where
n represents the sample size. Therefore, since this analysis has 89 independent variables, the
minimum sample size is 91 cases. Meyers et al (2006) also states that although this is the
minimum number needed in the smallest group (ADHD group in this case), it is not
recommended. The most common recommendation for determining sample size comes from
Tabachnick and Fidell (2007). They also confirm that the smallest group should exceed the
number of predictor variables (in this case 89); however, they state that a power analysis should
be conducted similarly to a MANOVA, given that the discriminant function analysis is
essentially a reverse MANOVA. Given this, a G-Power analysis was conducted for this analysis
with a Test of MANOVA, Global Effects was used given the discussion earlier from Tabachnick
and Fidell (2007) stating that the sample size for a Discriminant Function Analysis should follow
that of a simple MANOVA. Using a projected effect size of 0.25 and an alpha error probability
of 0.05, a Power value of 0.8, G-Power indicates that 218 participants are needed. In this
analysis, 596 participants were collected, surpassing the total indicated by the power analysis.
While a significantly larger sample size was used in this analysis, the two groups (ADHD or
Control) did not have an equal n. Therefore, since there were a greater number of control
68
participants, the probability of the model predicting someone to be in the control group was
almost four times as likely. Given this, SPSS has a feature compute from group sizes. This uses
the observed group sizes in the sample to define the prior probabilities of group membership,
statistically controlling for this uneven sample.
The discriminant function (the linear equation in the discriminant analysis) attempts to
maximally differentiate the two groups (ADHD and non-ADHD) on the independent variables
(test items). The prediction score is calculated based on the prediction weight (similar to a beta
weight in multiple regression) (Meyers, Gamst, & Guarino, 2006). The function is represented
as:
Di = a + bi Xi + b2X2 +…+ bn-Xn
The X’s are the predictor variables (items) and the b’s are the beta weights. The discriminant
function analysis uses the best fit method of maximum likelihood, rather than a least squares
solution as in a regression. This is an iterative process that determines the best fit (Meyers,
Gamst, & Guarino, 2006).
To interpret the outcomes, the structure matrix must be analyzed. The discriminant
loading (correlations between the variables and the discriminant function) should be at least .40
(Meyers, Gamst, & Guarino, 2006) to be considered a variable that discriminates between the
two groups. This will also include sensitivity and specificity. The analysis looks at which items
in general were the best predictors of ADHD in the college student population. The study also
analyzed how the new model found above predicts group membership compared to the original
11-item ADHD-EF Index.
Research Question 4. Is the factor structure of the BDEFS the same for college students
as it is for the normative sample, based on a confirmatory factor analysis?
69
The final research question addresses whether the factor structure of the BDEFS is the
same in college students as it was for Barkley’s normative sample. A confirmatory factor
analysis is the most suitable statistical procedure to answer this research question. As discussed
in the literature review, there are many different types of factor analyses, and the two primary
categories are exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) Brown,
2006; Meyers, Gamst, & Guarino, 2006). Because Barkley (2011b) has already conducted an
EFA and CFA on the multiple samples of adults, there is no need to conduct another EFA.
Moreover, there is theoretical and empirical support for the proposed factor structure. The CFA
is useful to determine whether a different group (college students) has the same factor structure
as the original group from the normalization sample.
To evaluate the proposed research question, the statistical software M-Plus 7.0 was
utilized. The data was converted to an MPlus file. There are five steps in evaluating a structure
model: “(a) model specification, (b) model identification, (c) model estimation, (d) model
evaluation, and (e) model respecification” (Meyers, Gamst, & Guarino, 2006, p. 549).
The sample size needed to conduct a CFA is difficult to determine. Experts in the field
do not necessarily agree on a standardized practice. According to Brown (2006), “Many rules of
thumb have been offered, including minimum sample size (e. g., N ≥ 100 to 200), minimum
number of cases per each freed parameter (e. g., at least 5 to 10 cases per parameter), and
minimum number of cases per indicator in the model (cf. Bentler & Chou, 1987; Boomsma,
1983; Ding, Velicer, & Harlow, 1995; Tanaka, 1987). Such guidelines are limited by their poor
generalizability to any given research data set. That is, the models and assumptions used in
Monte Carlo studies to provide sample sized guidelines are often dissimilar to the types of
models and data used by the applied researcher. Indeed, requisite sample size depends on a
70
variety of aspects such as the study design (e. g., categorical, continuous) and distribution of the
indicators, estimator type (e. g. ML, robust ML, WLSMV), the amount and patterns of missing
data, and the size of the model (model complexity).” P. 412-413. A more simplistic
recommendation was given by Tabachnick and Fidell (2007). They set their guide as “50 as very
poor, 100 as poor, 200 as fair, 300 as good, 500 as very good, and 1000 as excellent.” Therefore,
this researcher aimed for a sample of approximately 500-600 participants. While approximately
600 participants were collected, the sample size was again heavily skewed in favor of the control
group. The analysis was run twice, once with the entire sample and once with a sample in which
the control group was randomly reduced to have an equal number of participants in each group
(ADHD vs. Control). The random reduction was dine to more closely mimic Barkley’s first
CFA which was conducted on only the ADHD group, and Barkley’s second CFA conducted on
the norming sample. Given that there was a statistically significant difference between the
genders of the two groups, the female group was randomly reduced to equalize the two groups
on this variable. Results for both samples can be found in Table 13.
The most regularly used goodness-of-fit index is chi-squared (χ 2), which under the
classic maximum likelihood (ML) estimation model, is represented as: χ 2 = FML (N-1). In
MPlus, it is represented as χ 2 = FML (N). Even though χ 2 is the traditional model of ML, it is
rarely used on its own because it has some flaws (Brown, 2002). Primarily, it is susceptible to
distortions with high sample sizes and non-normally distributed samples. Given this, many
alternative fit indices have been developed to evaluate model fit and will be utilized in this study.
Table 2 lists various fit indices and their suggested values to indicate a good-fitting model.
Brown (2002) suggested that the research community is divided about the most appropriate
71
indices to utilize in CFA, and a bit of controversy surrounds it. He suggested three different
types of indices: absolute fit, parsimonious fit, and comparative fit.
Table 2 Suggested Ranges for Fit-Indices
Fit Indices Suggested ranges Reference X2 P≤ .05 Brown (2002)
X2/df < 2
Tabachnick & Fidell (2007)
CFI
< .90
Hu & Bentler (1999), Tabachnick & Fidell (2007)
RMSEA <.1
Not to exceed .06 <.05=good-fit, between .05-.08= reasonable-fit, <.08= poor-fit
Tabachnick & Fidell (2007) Taylor & Pastor (2007) Hu & Bentler (1999)
TLI < .95
< .90
Tabachnick & FIdell (2007) Hu & Bentler (1991)
SRMR ≤ .08 Tabachnick & Fidell (2007), Taylor &
Pastor (2007)
Absolute fit indices include χ 2, standardized root mean square residual (SRMR), root mean
square (RMR) (Brown, 2002), and goodness-of-fit (GFI) (Meyers, Gamst, & Guarino, 2006).
The analysis used in this study was the SRMR. Parsimonious indices are different from the
absolute fit indices because they have a consequence for poor model parsimony. The most
frequently used index of this type is root mean square error of approximation (REMSA) (Brown,
2002) and this is used in this analysis. In addition, the comparative fit index (CFI) and the
Tucker Lewis Index (TLI), also known as the non-normed fit index (NNFI) are used in this
study.
72
CHAPTER 4
RESULTS
Demographic Variables and Statistics
Demographic statistics were reviewed for all data collected and are represented
previously in Table 1. There were statistically significant differences between the ADHD
group and the Non-ADHD group for the variable Age (t = -3.73, p=.00). On average, participants
in the ADHD group were one year older than participants in the non-ADHD group. Chi-Square
tests were conducted to compare the two groups (ADHD verses non-ADHD) with regard to gender
(X2=6.44, p=.01), year in school (X2=.66, p=.88), and ethnicity (X2=4.96, p=.29), of which only
gender was statistically significant. The non-ADHD group had significantly more women than the
ADHD group.
Research Question 1
What is the relationship between the BDEFS self-report and other-informant report in a
college student population of students with ADHD? What are these relationships on the
following factors: Self-Management to Time, Self-Organization and Problem Solving, Self-
Restraint, Self-Motivation, Self-Regulation, ADHD-EF Index, and Total Executive Functioning
Symptoms? How do these correlations compare to the correlations that Barkley found in his
original study? Are the means of the self-informant reports higher or lower than the means of
the other-informant reports within the college student population of students with ADHD?
To evaluate the relationship between the self-report form (BDEFS) and the other-
informant report form (BDEFS-other), Pearson Correlations were conducted for the following
variables: the five factors, total score, and ADHD-EF Indexes. Results can be found in Table 3.
The correlations between BDEFS-self and BDEFS-other from Barkley’s analyses are listed in
73
Table 4 for comparison (Barkley, 2012b). A Fisher r to z transformation was conducted on four
of the factors to see if the results found in this study were significantly different from the results
found by Barkley. Recall that Barkley’s data did not have the Self-Regulation of Emotion, and
Barkley did not give inter-rater agreement correlation rates for the Total Score or the original 11-
item ADHD-EF Index, so these comparisons could not be made using the Fisher r to z
transformation. Results showed the following: Self-Management to Time z = -7.55, p =.00; Self-
Organization/Problem Solving z = -3.35, p=.00; Self-Restraint (z = -2.34, p =.019); and Self-
Motivation (z = -5.95, p = .00). These statistics indicate that all BDEFS-self/BDEFS-other
correlations in the current study are significantly different from the correlations reported by
Barkley. All of the comparable Barkley self/other correlations are higher than the current
self/other correlations.
Table 3 Inter-Rater Correlations for College Student Sample,
Comparing Self-Reports to Other-Reports
Factor r2 p
Self-Management to Time .22 .00 Self-Organization/Problem Solving .42 .00 Self-Restraint .39 .00 Self-Motivation .35 .00 Self-Regulation of Emotion .51 .00 Total Score .38 .00 Original 11-item ADHD-EF Index .80 .00 New 15-item ADHD-EF Index .24 .02
Table 4 Inter-Rater Correlations for Barkley’s Sample,
Comparing Self-Reports to Other-Reports
Factor r2 p
Self-Management to Time .79 < .00 Self-Organization/Problem Solving
.66 < .00
Self-Restraint .74 < .00 Self-Motivation .69 < .00
74
Following these analyses, a series of t-tests were conducted to compare the means of the
BDEFS-self and the BDEFS-other, in the current sample. These results were significant and can
be found in Table 5.
Table 5 Means, t-Tests, and p-Value for Self vs. Other-Informant Forms, in the
Current Sample
Means Self Other t p
Self-Management to Time 57.78 50.22 49.09 .00 Self-Organization/Problem Solving 60.59 50.03 46.06 .00 Self-Restraint 39.54 38.01 34.78 .00 Self-Motivation 26.46 22.44 35.53 .00 Self-Regulation of Emotion* 26.45 29.97 30.01 .00 Total Score 217.59 205.60 - - ADHD-EF Index 28.66 26.62 - -
Research Question 2
In the college student ADHD sample, is there a correlation between intellectual
functioning (as measured by the BIA) and the BDEFS similar to the correlation between the
intellectual functioning and BDEFS in the norming sample?
To evaluate the relationship between the Brief Intellectual Functioning (BIA) and the five
BDEFS factors, the original 11-item ADHD-EF Index, the new 15-item ADHD-EF index, and
Total Score; Pearson correlations were conducted. Please refer back to pages 75-78 for more
details regarding this analysis. The average Brief Intellectual Index (BIA) was 101.9 with a
standard deviation of 10.9. The correlations between the BIA and each of the factors were as
follows: Self-Management to Time (r=.26, p=.00); Self-Organization/Problem Solving (r=.04,
p=.64); Self-Restraint (r=.09, p=.28); Self-Motivation (r=.12, p=.16); Self-Regulation of
Emotion (r=.13, p=.15); Total Score (r=.15, p=.07); original 11-item ADHD-EF Index (r=.20,
p=.02), and new 15-item ADHD-EF index (r=.16, p=.06). In sum, the BIA was significantly
75
correlated with only two factors: Self-Management to Time and the original 11-item ADHD-EF
Index.
Research Question 3
Are the same ADHD-EF Index questions the most predictive of a diagnosis of ADHD in a
college student population as they are in the original normative sample?
To evaluate whether the same items from the original 11-item ADHD-EF Index were
needed to discriminate between students with and without ADHD in the college student
population, a control sample was obtained to compare to the clinic sample of students (a
description of this sample can be found in the Preliminary Analyses section, and is represented in
Table 1).
Models
In order to determine which items on the BDEFS best discriminate between college
students with and without ADHD, a discriminant function analysis (DFA) was conducted. First,
a DFA was conducted on all 89 items on the BDEFS. The DFA was then rerun with only the
items with a Structure Matrix loading of .4 or higher, which was the preset cutoff. There were
14 items at or over a Structure Matrix loading of .4. In addition, five items approached the .4
mark (.394-.372). Table 6 is a review of the canonical coefficients and structure coefficients of
each item with the items in bold to denote highest coefficients.
76
Table 6 Summary of Canonical Coefficients and Structure Loadings
Predictor (Scale Item) Canonical Coefficient
Structure Loadings
1 Procrastinate or put off doing things until the last minute -.145 .233
2 Poor sense of time -.009 .352
3 Waste or mismanage my time -.245 .315
4 Not prepared on time for work or assigned tasks -.223 .372
5 Fail to meet deadlines for assignments .094 .363
6 Have trouble planning ahead or preparing for upcoming
events. -.045 .375
7 Forget to do things I am supposed to do .149 .436
8 Can't seem to accomplish the goals I set for myself .083 .423
9 Late for work or scheduled appointments .233 .366 10 Can't seem to hold in mind things I need to remember to do .018 .426
11 Can't seem to get things done unless there is an immediate deadline
-.019 .357
12 Have difficulty judging how much time it will take to do something or get somewhere
.135 .417
13 Have trouble motivating myself to start work -.027 .298 14 Have difficulty motivating myself to stick with my work
and get it done .173 .402
15 Not motivated to prepare in advance for things I know I am supposed to do
-.122 .337
16 Have trouble completing one activity before starting into a new one
.020 .394
17 Have trouble doing what I tell myself to do -.042 18 Difficulties following through on promises or commitments
I may make to others .173 .364
19 Lack self-discipline .044 .299
20 Have difficulty arranging or doing my work by its priority or importance; can't "prioritize" well .318 .469
21 Find it hard to get started or get going on things I need to get done .223 .445
22 I do not seem to anticipate the future as much or as well as others
-.137 .291
23 Can't seem to remember what I previously heard or read about
-.124 .348
24 I have trouble organizing my thoughts .394 .527
77
Table 6 continued Predictor (Scale Item) Canonical
Coefficient Structure Loadings
25 When I am shown something complicated to do, I cannot keep the information in mind so as to imitate or do it correctly
-.083 .347
26 I have trouble considering various options for doing things and weighing their consequences
-.132 .325
27 Have difficulties saying what I want to say -.068 .244
28 Unable to come up with or invent as many solutions to problems as others seem to do
-.150 .210
29 Find myself at a loss for words when I want to explain something to others
-.121 .269
30 Have trouble putting my thoughts down in writing as well or as quickly as others
.114 .291
31 Feel I am not as creative or inventive as others of my level of intelligence -.216 .110
32 In trying to accomplish goals or assignments, I find I am not able to think of as many ways of doing things as others
-.100 .258
33 Have trouble learning new or complex activities as well as others
-.179 .296
34 Have difficulty explaining things in their proper order or sequence
-.094 .360
35 Can't seem to get to the point of my explanations as quickly as others .205 .345
36 Have trouble doing things in their proper order or sequence -.117 .376 37 Unable to "think on my feet" or respond as effectively as
others to unexpected events .007 .235
38 I am slower than others at solving problems I encounter in my daily life
.090 .293
39 Easily distracted by irrelevant events or thoughts when I must concentrate on something
.091 .470
40 Not able to comprehend what I read as well as I should be able to do; have to reread material to get its meaning .241 .413
41 Cannot focus my attention on tasks or work as well as others
.164 .536
42 Easily confused .227 .420
43 Can't seem to sustain my concentration on reading,
paperwork, lectures, or work .287 .505
44 Find it hard to focus on what is important from what is not important when I do things
-.151 .381
45 I don't seem to process information as quickly or as accurately as others
-.013 .334
78
Table 6 continued Predictor (Scale Item) Canonical
Coefficient Structure Loadings
46 Find it difficult to tolerate waiting; impatient .140 .303 47 Make decisions impulsively -.242 .247 48 Unable to inhibit my reactions or responses to events or
others -.194 .269
49 Have difficulty stopping my activities or behavior when I should do so.
.126 .301
50 Have difficulty changing my behavior when I am given feedback about my mistakes.
.051 .349
51 Make impulsive comments to others. .162 .270 52 Likely to do things without considering the consequences
for doing them. .051 .276
53 Change my plans at the last minute on a whim or last minute impulse.
-.060 .292
54 Fail to consider past relevant events or past personal experiences before responding to situations (I act without thinking).
.065 .288
55 Not aware of things I say or do. .028 .285 56 Have difficulty being objective about things that affect me. -.199 .166 57 Find it hard to take other people's perspectives about a
problem or situation. -.249 .098
58 Don't think or talk things over with myself before doing something.
.147 .311
59 Trouble following the rules in a situation. .164 .293 60 More likely to drive a motor vehicle much faster than others
(Excessive speeding). -.120 .102
61 Have a low tolerance for frustrating situations .156 .284 62 Cannot inhibit my emotions as well as others. .027 .202 63 I don't look ahead and think about what the future outcomes
will be before I do something (I don't use my foresight). .206 .275
64 I engage in risk taking activities more than others are likely to do.
.014 .207
65 Likely to take short cuts in my work and not do all that I am supposed to do.
.091 .352
66 Likely to skip out on work early if my work is boring to do. -.059 .301 67 Do not put as much effort into my work as I should or than
others are able to do. -.006 .321
68 Others tell me that I am lazy or unmotivated. -.082 .235 69 Have to depend on others to help me get my work done. .195 .314 70 Things must have an immediate payoff for me or I do not
seem to get them done. -.123 .313
79
Table 6 continued Predictor (Scale Item) Canonical
Coefficient Structure Loadings
71 Have difficulty resisting the urge to do something fun or more interesting when I am supposed to be working.
.053 .309
72 Inconsistent in the quality or quantity of my work performance.
.000 .370
73 Unable to work as well as others without supervision or frequent instruction.
.027 .323
74 I do not have the willpower or determination that others seem to have.
-.193 .327
75 I am not able to work toward longer term or delayed rewards as well as others. .254 .366
76 I cannot resist doing things that produce immediate rewards, even if those things are not good for me in the long run.
-.034 .283
77 Quick to get angry or become upset. -.149 .153 78 Overreact emotionally. -.045 .124 79 Easily excitable. -.313 .076 80 Unable to inhibit showing strong negative or positive
emotions. -.043 .154
81 Have trouble calming myself down once I am emotionally upset.
-.089 .142
82 Cannot seem to regain emotional control and become more reasonable once I am emotional.
-.139 .130
83 Cannot seem to distract myself away from whatever is upsetting me emotionally to help calm me down. I can't refocus my mind to a more positive framework.
.286 .257
84 Unable to manage my emotions in order to accomplish my goals successfully or get along well with others. .200 .279
85 I remain emotional or upset longer than others. -.101 .155 86 I find it difficult to walk away from emotionally upsetting
encounters with others or leave situations in which I have become very emotional.
.013 .126
87 I cannot re-channel or redirect my emotions into more positive ways or outlets when I get upset.
-.093 .196
88 I am not able to evaluate an emotionally upsetting event more objectively.
-.103 .194
89 I cannot redefine negative events into more positive viewpoints when I feel strong emotions.
-.128 .191
Note: Canonical Coefficient items in boldface account for the highest importance for describing differentiation among groups. Structure loadings in boldface account for the highest amount of contribution to the significant discriminant function.
80
The model was run with these five items and combinations thereof to see if they increased the
sensitivity and specificity of the model. Only one of these items (item number 16 with a matric
loading of .394) improved the model; therefore, item number 16 was included in the model.
These 15 items are listed in Table 7 with a description and the factor in which they belong.
Table 7 New 15-Item ADHD-EF Index
# Item description Factor r
7 Forget to do things I am supposed to do Time .149
8 Can’t seem to accomplish the goals I set out for myself Time .083
10 Can’t seem to hold in mind things I need to remember to do Time .018
12 Having difficulty judging how much time it will take to do something or get somewhere
Time .065
14 Having difficulty motivating myself to stick with my work
and get it done
Time -.084
16 Have trouble completing one activity before starting into a
new one
Time -.074
20 Having difficulty arranging or doing my work by its priority or importance; can’t “prioritize” well
Time .238
21 Find it hard to get started or get going on things I need to get done
Time .066
24 I have trouble organizing my thoughts Organization .292
39 Easily distracted by irrelevant events or thoughts when I must concentrate on something
Organization .120
40 Not able to comprehend what I read as well as I should be able to do; have to reread material to get its meaning
Organization .146
41 Cannot focus my attention on tasks or work as well as others Organization .189
42 Easily confused Organization .053
43 Can’t seem to sustain my concentration on reading, paperwork, lectures, or work
Organization .167
Note: Bolded items overlap with the items found on the original 11-item ADHD-EF Index
81
The original 11-item ADHD-EF Index items are listed in Table 8 for comparison. Three of these
items overlap between the models and are identified in bold. The new 15-item ADHD-EF Index
had the ability to significantly discriminate between the ADHD group and the control group, and
this model represented the highest degree of specificity and sensitivity than the other models
attempted. The model is presented in Table 9 along with several other models run for
comparison (described below). The new 15-item ADHD-EF Index accounted for 44.8% of the
total relationship between the items and diagnosis. In addition to the new 15-item model
represented previously, several separate DFA’s were run using different sets of items. The
different models are described in Table 10.
Group Centroids
The group centroids for the four models are represented in Table 11. The group centroid
for the new 15-item ADHD-EF Index discriminates the most between the ADHD group and the
non-ADHD group.
Table 8
Original 11-Item ADHD-EF Index
# Item Description Factor
1 Procrastinates or puts things off until the last minute Time
6 Have trouble planning ahead or preparing for upcoming events Time
14 Having difficulty motivating myself to stick with my work and get it
done
Time
16 Having trouble completing one activity before starting into a new
one
Time
24 I have trouble organizing my thoughts Organization
49 Having difficulty stopping my activities or behavior when I should do so Restraint
82
Table 8 continued
Item Description Factor
50 Having difficulty changing my behaviors when I am given feedback about my mistakes
Restraint
55 Not aware of things I say or do Restraint
60 More likely to drive a motor vehicle much faster than others Restraint
65 Likely to take short cuts in my work and not do all that I am supposed to do
Motivation
69 Have to depend of others to help me get my work done Motivation
Note: Bolded items overlap with the items found on the new 15-item ADHD-EF Index
Table 9 Summary of Canonical Discriminant Functions
ADHD-EF Index Eigenvalue
% Variance
Canonical Correlation
R*
Canonical R2
Lambda Chi-
Square df Sig
New 15-item
.809 100.00 .669 .448 .553 330.65
1 14 .00
Original 11-item .660 100.00 .630 .397 .603
283.425
11 .00
5-item .602 100.00 .613 .376 .624
264.997
5 .00
2-item .636 100.00 .623 .388 .611
277.562
2 .00
Classification Rates
As far as overall classification rate, the highest rate was found with the new 15-item
ADHD-EF Index. The new 15-item ADHD-EF Index also had a higher sensitivity rate.
However, the specificity rate was relatively equal across all models. The classification data for
each model are represented in Table 12.
83
Table 10 ADHD-EF Index Description of Models
Model Description
Original 11-item ADHD-EF Index Barkley’s 11-item model which is published in the BDEFS manual (Barkley, 2011b).
New 15-item ADHD-EF Index The 15 items with the best discrimination between the ADHD and Non ADHD groups determined in this study.
5-item community control ADHD-EF Index The five items that Barkley’s logistic regression yielded from comparing the community control sample with the ADHD sample in his study. These five items are included in the 11-item model.
2-item ADHD-EF Index (screening tool) The two-item screening model was derived by using the two items with the highest canonical correlations from this study.
Table 11 Functions at Group Centroids
Function
Group New 15-
item Original 11-item
Community Control 5-
item
2-item (screener)
Control Group -.436 -.393 -.376 -.386 ADHD Group 1.851 1.671 1.596 1.641
Table 12 Classification Rates
Model Overall Sensitivity Specificity
New 15-item ADHD-EF-Index
91 81.5 93.2
Original 11-item ADHD-EF Index
89.1 69.4 93.7
Community Control 5-item model
87.7 63 93.5
2-item (screener) 88.4 68.5 93
84
Research Question 4
Is the factor structure of the BDEFS the same for college students as it is for the
normative sample, based on a confirmatory factor analysis?
The CFA was conducted using the modeling software M-Plus 7 on the 89-items of the
BDEFS. The hypothesized model is the model presented by Barkley (2011b) in that items 1-21
belong to Factor 1 (Self-Management to Time), items 22-45 belong to Factor 2 (Self-
Organization/Problem Solving), items 45-64 belong to Factor 3 (Self-Restraint), items 65-76
belong to Factor 4 (Self-Motivation), and items 77-89 belong to Factor 5 (Self-Regulation of
Emotions). This five-factor model is hypothesized to be the same in the college student
population as it was in Barkley samples. The five factors are hypothesized to covary with each-
other. The assumptions of multivariate linearity and normality were reviewed through SPSS.
Maximum Likelihood estimation was used to estimate the models. The model was analyzed on
the full sample of 596 and on the reduced model of 310 (to match Barkley’s clinic sample). For
both of the models, the chi square, comparative fit index (CFI), root mean square error of
approximation (REMSA), Tucker Lewis index, and standardized root mean square residual were
calculated.
Barkley first used an EFA model on his clinic referred sample of adults. He then tested
the CFA on the same clinic referred sample and then on the normative sample. The full sample
in this study most closely matches the normative sample. The reduced sample in this study most
closely matched his clinic referred sample; therefore, results from both samples were compared
and are represented in Table 1. The hypothesized model was tested and there was reasonable
support for Barkley’s five factor model in both samples.
85
Table 13 CFA Models
Model X2 df X2/df CFI RMSEA TLI SRMR p
N=595 10467.201 3817 2.742 .840 .054 .836 .055 .000 N=310 8412.733 3817 2.204 .801 .062 .796 .062 .000
In comparing the models, the suggested fit indices listed in Table 2 are utilized. After
comparing the two models, it was determined that both models were similar in characteristics.
Tabachnick and Fidell (2007) indicate that a standardized factor loading to indicate good fit
should be a value of .6 or above. The standardized factor loadings are represented in Table 14.
However, Comrey and Lee (1992) suggest that any factor loading above .55 is “good.” If using
the Comrey and Lee suggestions, only two of the eighty-nine factor loadings had a fit less than
“good.” If using Tabachnick and Fidell’s (2007) slightly more conservative fit cut-off, only
seven of the eighty-nine factor loadings were not in the acceptable range. The correlations for all
factors were significant (p=.000) and are represented in Table 14 as well.
Table 14 Standardized Factor Loadings and Standardized Residual Variances
Factor/Item Number
STDYX p -value
Factor 1 EF1 .667 0.00 FF2 .698 0.00 EF3 .794 0.00 EF4 .782 0.00 EF5 .700 0.00 EF6 .779 0.00 EF7 .737 0.00 EF8 .792 0.00 EF9 .668 0.00 EF10 .761 0.00 EF11 .793 0.00 EF12 .745 0.00 EF13 .776 0.00 EF14 .852 0.00 EF15 .822 0.00 EF16 .782 0.00
86
Table 14 continued Factor/Item Number
STDYX p -value
Factor 1 EF17 .799 0.00 EF18 .696 0.00 EF19 .743 0.00 EF20 .771 0.00 EF21 .814 0.00
Factor 2
EF22 .611 0.00 EF23 .740 0.00 EF24 .817 0.00 EF25 .690 0.00 EF26 .702 0.00 EF27 .673 0.00 EF28 .687 0.00 EF29 .734 0.00 EF30 .641 0.00 EF31 .478 0.00 EF32 .730 0.00 EF33 .726 0.00 EF34 .783 0.00 EF35 .774 0.00 EF36 .783 0.00 EF37 .683 0.00 EF38 .712 0.00 EF39 .753 0.00 EF40 .703 0.00 EF41 .808 0.00 EF42 .783 0.00 EF43 .780 0.00 EF44 .778 0.00 EF45 .782 0.00
Factor 3 EF46 .544 0.00 EF47 .741 0.00 EF48 .778 0.00 EF49 .710 0.00 EF50 .716 0.00 EF51 .767 0.00 EF52 .793 0.00 EF53 .635 0.00 EF54 .797 0.00 EF55 .712 0.00
87
Table 14 continued Factor/Item Number
STDYX p -value
Factor 3 EF56 .724 0.00 EF57 .557 0.00 EF58 .734 0.00 EF59 .663 0.00 EF60 .459 0.00 EF61 .655 0.00 EF62 .676 0.00 EF63 .704 0.00 EF64 .585 0.00
Factor 4
EF65 .818 0.00 EF66 .760 0.00 EF67 .755 0.00 EF68 .640 0.00 EF69 .691 0.00 EF70 .811 0.00 EF71 .693 0.00 EF72 .785 0.00 EF73 .695 0.00 EF74 .744 0.00 EF75 .806 0.00 EF76 .745 0.00
Factor 5 EF77 .705 0.00 EF78 .783 0.00 EF79 .555 0.00 EF80 .642 0.00 EF81 .820 0.00 EF82 .827 0.00 EF83 .805 0.00 EF84 .812 0.00 EF85 .787 0.00 EF86 .700 0.00 EF87 .815 0.00 EF88 .836 0.00 EF89 .834 0.00 Note: Structure loading in bold indicate items under the structure loading cut-off of .6.
88
CHAPTER 5
DISCUSSION
The aim of this study was to investigate the psychometric properties of the Barkley
Deficits in Executive Functioning Scale (BDEFS) in a college student population. The BDEFS
was published in 2011, and there has yet to be independent support for the psychometric
properties of this measure. The BDEFS is an 89-item self-report scale used to measure
impairment in executive functioning. Barkley’s research suggests that the BDEFS is a five-
factor model. This study analyzed the self-informant vs. other-informant correlations for the five
factors of the BDEFS. Additionally, this study analyzed the relationship between the five
BDEFS factors and a measure of brief intellectual ability. In addition, a discriminant function
analysis was conducted to see which BDEFS items best discriminated between participants with
and without a diagnosis of ADHD in a college student population. Finally, a confirmatory factor
analysis was conducted to see if the factor structure of the BDEFS in a college population was
similar to the original factor structure.
This chapter discusses the results of the study and conclusions of the analyses conducted.
The findings are organized by research question and the practical implications for these findings
are discussed. Finally, limitations of the study are analyzed.
Relationship between BDEFS Self-Report Form and Other-Informant Form
This research question had multiple goals: (a) what is the relationship of the self-
informant ratings and the other-informant ratings on the five factors of the BDEFS, (b), are these
correlations statistically significantly different from the correlations that Barkley found, and (c)
are the self-informant ratings higher or lower than the other-informant ratings. The current study
identified significant self-informant rating and other-informant rating correlations for all BDEFS
89
factors. It is interesting that the factors which include more outward behaviors (Self-Restraint
and Self-Regulation) were correlated highest, and the factors which include more inward
behaviors were more weakly correlated (Self-Management to Time, Self-Motivation, and Self-
Organization/Problem Solving). Specific demands of the college student which are often
different from the general adult population rely heavily on the inward behaviors; therefore, the
college student may feel more pressure in these areas and rate themselves more impaired than
others would rate them. The college student with ADHD often is able to reach deadlines, but
procrastinates until the final moments. They often become exceedingly frustrated with their time
management and motivation issues (rating themselves more impaired on those BDEFS question);
however, others may not see this struggle (rating them less impaired).
In looking at how the correlations in this study compared to the correlations in the study
Barkley conducted, it was found that all BDEFS self-ratings and BDEFS other-ratings
correlations in the current study were significantly lower than the correlations identified by
Barkley. We can speculate as to why the correlations between self- and other-ratings were lower
in this sample than in Barkley’s sample. One, many of the questions on the BDEFS are about
symptoms that cannot necessarily be seen by another individual. For example, “wastes or
mismanages time” (item 3), “has trouble motivating self to work” (item 13), “has difficulty
saying what he/she wants to say” (item 27), “or cannot focus on the task at hand as well as
others” (item 41) may be particularly difficult for someone else to rate. On the original study
published by Barkley (211b), correlations between self-informants and others-informants were
strong at .66-.79. The correlations found in this study were weak to moderate (.295-.534), and
statistically significantly smaller than those found by Barkley. The only information about the
other-informant given from Barkley’s analysis was that this should be a person who knows the
90
person well. In the present study, this was often a significant other, parent, or roommate. As was
just discussed, a reason that the college students’ self-ratings may have varied more from the
other-informant ratings versus the general adult population is perhaps due to the difference in
demands placed on the two different groups. The responsibilities in the college setting tend to be
a greater reliance on executive functions than in the adult in general. Given that the majority of
these are unseen by others, it is not improbable that these results were found.
In addition, it was found in the current study that the BDEFS self-ratings were higher, in
all but one case, than the BDEFS other-ratings. The BDEFS self-ratings were not higher than the
BDEFS other-ratings for the factor Self-Regulation of Emotion. In evaluating these results, the
researcher must consider several possibilities. Are differences in self-other ratings scores a
function of the scale itself or a function of the rater? Several possibilities can be speculated as to
why the self-ratings tended to be higher than the other-ratings. First, in the case of college
students the risk of malingering must be considered. As was discussed in the literature review,
stimulant medications such as used in the treatment of ADHD have been misused and abused at
alarming rates on college campuses. Students may try to look worse than they are in order to
obtain medication (Booksh, et al., 2010). Second, related to this issue, is that of “crisis”.
Students who are referred to this clinic for evaluation of their symptoms of ADHD are generally
doing so because they are in some kind of crisis situation, such as failing a class or academic
probation. Therefore, they are likely to rate themselves as more impaired than would their other-
informant.
Relationship between the BIA and BDEFS Factors
The second area of research was aimed at examining the relationship between the
intellectual ability of college students with ADHD and their responses on the BDEFS. This is an
91
important analysis, in that it addresses a major limitation of traditional EF tests. The traditional
EF tests were often contaminated by IQ, specifically indices of intelligence including measures
of motor and naming speed, which is not related to EF (Salthouse (1996, 2005). In addition,
there is a significant overlap between IQ and EF tests in relation to working memory (Antshel et
al., 2010). The current study found that intelligence was significantly correlated with one BDEFS
factor: Self-Management. This relationship was in the positive direction (meaning as IQ
increased, so did the level of impairment in Self-Management). Additionally, the correlation was
relatively weak (r = .26). There are several plausible explanations for the correlation between
intelligence and the factor Self-Management to Time. The first issue is that of selection bias. All
participants in this analysis were non-randomly selected, in that they were either self-referred or
referred by a doctor, counselor, or academic advisor for evaluation of ADHD. Furthermore, all
participants received a diagnosis of ADHD following their evaluation. As was discussed in the
literature review, college students with ADHD tend to have some protective factors to help them
succeed in their academic programs. Once such protective factor may be intelligence (DuPaul et
al., 2009; Glutting et al., 2005). High school students who have ADHD (whether known or
unknown) and continue on to college after graduation likely do so based on their intellectual
abilities, rather than organizational skills. More specifically, the skill of time management may
have never been developed. Put succinctly, students with intellectual capacity sufficient to
complete high-quality work at the last minute with minimal organization may never have been
forced to organize their workload in high school. Students with ADHD (i.e. poor time
management skills) who did not have the requisite intellectual functioning likely did not self-
select to attend college. Therefore, they are not represented in this sample. In essence, the
higher the intellectual ability, the less the need for developing time management skills to succeed
92
in high school. This may partially explain why, as intelligence increased, so did the impairment
of time management. In a student with both ADHD and high IQ, the degree of intellectual
ability might actually function to progressively reduce the consequences for failing to learn basic
time management strategies, resulting in an inverse correlation between these areas.
An alternative (and complementary) way to view this correlation is that a college student
who has high intelligence and also has good time management skills (whether diagnosed with
ADHD or not) would be unlikely to find themselves being evaluated for ADHD. Mostly
students in “crisis” situations or struggling in some way were included in this sample. On an
item level, the types of information gathered in the factor of Self-Management to Time relate
heavily to procrastination and self-discipline relative to school work, which is consistent with
research on salience and ADHD. It has been noted anecdotally and supported by research
(Zentall, 2005), that students with ADHD tend to become most productive when a deadline is
approaching. Again, students who procrastinate until the last minute and do not have the ability
to produce good work at the last minute (e.g. students with lower relative intellectual ability) are
less likely to find themselves in college. For higher ability individuals, college may represent the
first time that their intellectual abilities are overmatched by the need for time management,
leading to increased “crisis” situations and subsequent referrals for evaluation.
Barkley’s analysis utilizing an early version of the BDEFS found only the Self-
Organization/Problem Solving factor to be associated with IQ. The results of the current study
supported the researcher’s original hypothesis that Self-Organization/Problem Solving would not
be significantly correlated with IQ. While the prediction was borne out, it was predicated on the
assumption that the intellectual abilities in the college sample would be higher than the general
population. However, the Brief Intellectual Ability (BIA) in the current study was average
93
(X=101.9, SD=10.6). There are a few reasons why this may have been the case. First, Barkley
used a full scale IQ in his study, and the current study used the BIA. The BIA has only .60-.69
correlation with a full measure of IQ (McGrew & Woodcock, 2011), and clinical experience has
demonstrated that it is generally an underestimate of full scale IQ. Furthermore, the BIA is only
made up of three sub-tests. Two of these are heavily influenced by timed testing (Concept
Formation and Decision Speed) and one (Concept Formation) is consistent in design with tasks
recruiting working memory and attention to detail. Students with ADHD tend to struggle with
processing speed tasks (Weyandt, 2005), potentially demonstrating a lowered BIA than if a non-
timed or mixed measure were used. Given this, it remains unclear to what degree intelligence is
related to the Self-Management to Time factor. However, this would be consistent with the
hypothesis that utilizing the BIA yields suppressed IQ scores overall, while allowing for
reasonable comparison within the sample.
Reevaluation of ADHD-EF Index
The current 11-item ADHD-EF Index developed by Barkley was created to provide
clinicians with a brief tool to predict a diagnosis of ADHD in the adult population. Of the 89
items on the BDEFS, these 11 items were selected using two logistic regressions which is
discussed in detail in the literature review of this manuscript. While having such a brief index is
valuable in the clinical setting to identify individuals who are likely to have ADHD for the
purposes of referring them for a more extensive assessment, this index has not been validated on
a college student population. Therefore, the same items that predict ADHD in the adult
population may not be the best predictors of ADHD in a college student population. When
examining the ability of some items on the BDEFS to successfully discriminate between the
ADHD group and the non-ADHD group, results show that the best model was the new 15-item
94
ADHD-EF-Index which successfully categorizes 91% of the participants and accounted for
44.8% (canonical R2 = .448) of the variance in a college student population. When the original
11-item ADHD-EF Index was used on the current college student sample, the 11 items were able
to discriminate between those students with and without ADHD, but to a slightly lesser extent
(89%, canonical R2= .397). Interestingly, only three of the items from the original 11-item index
overlap with the new 15-item ADHD-EF Index, thus indicating that, in a college population,
different items are needed to accurately discriminate between those with ADHD and those
without.
The three items that overlap are all from the factors Self-Management to Time and Self-
Organization/Problem Solving, and these items are: “Having difficulty motivating myself to stick
with my work and get it done” (item 14, Self-Management to Time), “Having trouble completing
one activity before starting into a new one” (item 16, Self-Management to Time), and “I have
trouble organizing my thoughts” (item 24, Self-Organization/Problem Solving).
The original ADHD-EF Index scale uses 11 items to discriminate individuals with
ADHD from those without. Of the items that overlapped between the two scales, one was from
Barkley’s comparison of the ADHD group to his clinical control group, and two were derived
from his comparison between his ADHD and the community control group. When using the five
items from the original 11-item ADHD-EF Index that were obtained from the sample most
closely related to the sample in this study, these five items (1, 16, 24, 50, and 65) discriminate
almost as well as the original 11-item ADHD-EF Index (at 87.7% vs. 89%). Therefore, when
looking at the college student population, the original 11-item ADHD-EF Index may not be the
most efficient model when attempting to discriminate students with and without ADHD. In fast-
95
paced clinical settings, gaining approximately one percentage point of discriminant ability may
not justify a greater than 100% increase in the length of the screening instrument.
In looking at the difference between the items on the original 11-item ADHD-EF Index
and the new 15-item ADHD-EF Index, there are some general themes. On the original 11-item
ADHD-EF Index, there are four items from the factor Self-Management to Time, one from Self-
Organization/Problem Solving, four from Self-Restraint, and two from Self-Motivation. The
new scale pulled only from Self-Management to Time (eight items) and Self-
Organization/Problem Solving (six items) all of which are listed below.
Self-Management to Time
Forget to do things I am supposed to do (item 7)
Can’t seem to accomplish the goals I set out for myself (item 8)
Can’t seem to hold in mind things I need to remember to do (item 10)
Having difficulty judging how much time it will take to do something or get somewhere
(item 12)
Having difficulty motivating myself to stick with my work and get it done (item 14)
Have trouble completing one activity before starting into a new one (item 16)
Having difficulty arranging or doing my work by its priority or importance; can’t
“prioritize” well (item 20)
Find it hard to get started or get going on things I need to get done (item 21)
Self-Organization/Problem Solving
I have trouble organizing my thoughts (item 24)
96
Easily distracted by irrelevant events or thoughts when I must concentrate on something
(item 39)
Not able to comprehend what I read as well as I should be able to do; have to reread
material to get its meaning (item 40)
Cannot focus my attention on tasks or work as well as others (item 41)
Easily confused (item 42)
Can’t seem to sustain my concentration on reading, paperwork, lectures, or work (item 43)
As discussed in the literature review, the population of college students with ADHD
tends to be either Combined Type ADHD or Inattentive Type ADHD (Barkley and Murphy,
2011). Far fewer of the Hyperactivity Type ADHD only is seen in the college population.
(DuPaul et al., 2009). In fact, none of the participants in the current sample were of the
Hyperactive subtype. There are several hypotheses of why this might be; however, it is widely
accepted that students either “grow-out” of these behaviors or the acts of being accepted into and
attending college self-selects for students with more self-control, successful academic histories,
advanced coping skills, and higher cognitive abilities (DuPaul et al., 2009; Glutting et al., 2005).
Thus, it is not surprising that a major difference between the original 11-item ADHD-EF Index
derived from an adult population has several items from the factor, Self-Restraint, and the model
derived from the college student population has no such items. Along the same lines, no items
from the factor Self-Motivation were noted to be derived from the college student population.
Again, college students are self-selected to be more motivated than the non-college student
(Reaser et al., 2007); therefore, items tapping these types of symptoms would not differentiate
between students with and without ADHD.
97
The time demands facing a college student may be more than that of non-college student
adults. Given the increased pressure in college to have good time management and
organizational skills, it is not unlikely that the items needed to differentiate the ADHD group
from the non ADHD group would be different in the college student population verses the adult
population. For example, when surveying most college students about procrastination and skills
related to planning-ahead, one might expect to find most students struggling with the perception
of difficulties in these areas. Therefore, items tapping these symptoms may not necessarily
distinguish between college students with and without ADHD as it does in the general adult
population. The results here are similar to what was seen by Murphy (2005) and Proctor and
Prevatt (2009) when they asserted that college students with ADHD have more problems
focusing, making deadlines, task completion, and sustaining effort in presumed irrelevant tasks.
Overall, when comparing the specificity rate (percentage of students without ADHD
correctly identified in the non-ADHD group) there were no clinically relevant differences,
indicating all of the models were equally effective in accurately classifying participants as non-
ADHD. However, there were clinically relevant differences with regards to sensitivity (the
ability to correctly identify someone with ADHD). The new 15-item ADHD-EF Index had a
sensitivity rate of 81.5% versus the original 11-item ADHD-EF Index (69.54%), the 5-item
community control ADHD-EF Index (63%), or the 2-item ADHD-EF Index screener (68.5%)
which will be discussed below. This provided evidence in support of using a different model to
predict or screen for ADHD in the college student population. If given as a screening device, the
new 15-item ADHD-EF Index has a much better rate of correctly identifying college students
who likely have ADHD. The clinical utility of a quick screening tool is substantial in a
university or college campus setting. Given that college students with ADHD experience
98
significantly more academic consequences than college students without ADHD (Reaser et al.,
2007), it is important to efficiently identify these students in order to offer treatment and
assistance. It is worth emphasizing that the ADHD-EF Index is meant to be a screening tool.
With this in mind, if error is present, it is desirable to err in the direction of over-identifying
students as belonging to the potential-ADHD group so that they can be referred for a full
diagnostic evaluation of their symptoms.
In addition to the new 15-item ADHD-EF Index and the 5-item Community Control
ADHD-EF Index, another interesting result came of these analyses. A very brief two-item model
identified items number 20 (having difficulty arranging or doing my work by its priority or
important; can’t prioritize well) and 24 (I have trouble organizing my thoughts) as generally
equal in specificity and sensitivity to the original 11-item ADHD-EF Index. Thus, these two
items alone may be quite beneficial as a quick screening tool. If a clinical or academic advisor
asks a student if they are having difficulty prioritizing their tasks and difficulty organizing their
thoughts, the likelihood of missing a student who has ADHD is only about 7%. Of course, this is
not diagnostic given that the sensitivity is in the high 60% range. However, this is a useful,
quick, and low cost screening method for determining which students to refer for more thorough
assessment.
Evidence of Factorial Validity
Only one researcher, the author of the BDEFS, has evaluated the factor structure of the
BDEFS to date. No study has yet been published looking at this factor structure for the BDEFS
on a college student population. The current study found moderate support for the published
factor structure of the BDEFS in the college student population. This support was found in both
samples analyzed in this study. While there was support for the factor structure, some of the fit
99
indices were not as strong as had been hoped for. As mentioned in the planned analysis section,
there are multiple ways to determine goodness of fit, or evaluating the model, for a CFA. One of
the most common is the Chi Squared (X2). A goodness of fit is represented by a non-significant
X2 and in this analysis, the X2 was significant. However, one of the short-falls with relying on X2
is the influence of a large n on the statistic making most any differences statistically significant
(Tabachnick & Fidell, 2007). Given the problems with sample size and the underlying
assumptions that influence the results, other statistics have been proposed. Related to the X2, a
cursory measure to look at fit is to use the ratio of X2 to the degrees of freedom. Referring back
to Table 2, if this is less than two, this provides support for the model (Tabachnick & Fidell,
2007). In this analysis, the X2 is 2.2 indicating a level just slightly over the generally accepted
range. The CFI and TLI fit indices were at the moderate level showing near their respective
specified cut off points. However, the larger sample (more closely aligned with the normative
sample) had fit indices values that were slightly stronger. The REMSA and the SRMR are the
two fit indices which show the best support for Barkley’s factor model of the BDEFS. These
indexes are best suited for larger samples (like the current sample) and may explain the reason
these indices were more supportive than the preceding indices that are heavily influenced by the
larger sample size. Another potential explanation for the moderate results relates to the
assumption that a sample must be normally distributed. This is an impairment rating scale.
Therefore, the participants in the control sample and the participants in the ADHD sample likely
rated the items quite differently, and their responses were not normally distributed. There is also
the possibility of a floor effect for the control sample.
When looking at the items on an individual level, all 89 items has a statistically
significant factor loading, meaning each item contributed to its respective factor indicating
100
evidence of validity to the factor structure of the BDEFS in a college student population. While
all items were significant, there were a few items that fell just under the .55 cut off score for
“good” fit. The two items were “I feel I am not as creative or inventive as others of my
intelligence level” (item 31) and “More likely to drive a motor vehicle faster than other” (item
60). While still statistically significant, these two items account for the least amount of variance
in the model. When looking at the item pertaining to creativity, the college student population
may differ on their perceptions of creativity as it relates to intelligence in a different way than the
general adult population does. Specifically, college students are generally higher in cognitive
abilities than the adult population and creativity may be more revered on college campuses as
well. As published, this item belongs to the factor Self-Organization/Problem Solving. This
question about creativity does not necessarily fit with the other type of items when referring to a
college student population. For instance, other items in this area focus on tasks of concentration
and organizing, with only one other questions discussing something similar to creatively (coming
up with a new way to solve a problem). The question pertaining to creativity may be viewed as a
difficult area in the college student population regardless of any impairment on that factor.
The item related to driving falls within the factor of self-restraint. Other items in this factor
pertain mostly to impatience and impulsivity. While driving fast is generally related to ADHD
and EF deficits, it is also common for this age group in general. Thus, endorsing this item may
not have much to do with the endorsement of other items included in this factor making it a less
than desirable fit.
Limitations
There are several issues that could affect the findings of the study. All participants in this
study were acquired from a large public university in the southeastern United States. The results
101
of this study may not generalize to other college student populations such as private colleges or
universities, community colleges, or universities in northern or western areas. In addition, these
students were mostly traditional students under the age of 30. The results found may not
generalize to non-traditional (or second career) students or graduate students. Students in this
study were recruited in several different ways, which could have affected the results. The
control group was primarily from one college of the university and was heavily female in
composition. To remedy this, additional control participants were collected from the same
university in several undergraduate classes and in general areas on campus. Participants in the
one college received either extra credit for participating or participation fulfilled a course
requirement. The participants recruited in other parts of the university were offered monetary
incentives for participating. These different methods could have influenced the way the
participants responded to the survey.
In regards to the participants in the ADHD group, there are several issues which could
have affected the validity of the results as well. As reported, the participants in the ADHD group
were evaluated at an on-campus clinic to determine whether or not they qualified for a diagnosis
of ADHD. Students often presented for an evaluation when they were in a crisis situation such
as academic probation or losing a scholarship due to poor academic performance. In addition,
students may have been looking for a diagnosis of ADHD to acquire stimulant medications
(secondary gain). All of these factors could have influenced the way in which they completed
the survey which measured impairment.
As far as the evaluation and determination of ADHD is concerned, all students who were
given a diagnosis met criteria for ADHD using the DSM-IV-TR. Since that time, the DSM-5 has
been published, with a slightly less stringent criteria for diagnosing ADHD in adults and with
102
more relevant criteria. Therefore, there were likely many students who did not receive a
diagnosis of ADHD and were excluded from this sample that would receive such a diagnosis if
currently evaluated It is possible that a sample with more moderate levels of ADHD could
influence the results. Finally, the students in the ADHD group were given the Brief Intellectual
Abilities Index (BIA) from the Woodcock Johnson Tests of Cognitive Abilities as a proxy for
IQ. Unfortunately, the BIA has only a .60-.69 correlation with IQ from other full measures of
intelligence (McGrew & Woodcock, 2011). This may explain why results from that research
question were moderate in nature.
Implications for Future Research
Given that the BDEFS is in its infancy, there are multiple avenues for continued research.
The Discriminant Function Analysis (DFA) in this study was conducted with sufficient sample
size; however, there were not enough participants in each group to run a split-half analysis. This
would entail conducting the DFA on half of the sample, then using the results from that analysis
to check the specificity and sensitivity on the other half of the data. This would increase the
validity of the results. In addition, given that the new 15-Item ADHD-EF Index has been
produced on this single sample, it is recommended that replication studies be conducted before
this is used in clinical practice as a standalone measure.
The control sample in the current study had a higher representation of females; therefore
a reduced sample was used for the CFA. It is advised that a sample of at least 500 participants,
with equal gender distribution, be collected to run the CFA again. In addition, a multi-sample
CFA should be conducted given the anticipated difference in these groups regarding impairment.
In continuing to investigate the psychometric properties of the BDEFS on a college student
population, several other analyses should be considered. This current study did not exclude
103
participants in the ADHD group who had co-occurring disorders. During the course of the
ADHD evaluation, participants complete a DSM checklist of clinical symptoms. These
symptoms should be correlated with the five factors of the BDEFS to see if there is a
relationship. As well, Barkley (2012b) noted that when there were higher internalizing
symptoms, the participant was more likely to rate themselves more impaired on the BDEFS than
their other informant. This should be replicated in the college population. Finally, concurrent
validity should be evaluated by comparing responses on the BDEFS to the Behavior Rating
Inventory of Executive Functioning (BRIEF-A), which has evidence of validity.
Implications for Clinical Practice
One of the most notable implications resulting from this study is the identification of the
new 15-item ADHD-EF Index that better discriminates college students with ADHD from
students without ADHD. In addition to this new scale, the two brief screening questions
identified as highly identifying students that may have ADHD could be used by campus
professionals to quickly identify students who may need a more extensive evaluation. These
professionals may include mental health counselors, academic counselors, and medical
professionals. Another finding indicating that the 5-item ADHD-EF Index (from the original
ADHD-EF Index) was also quite useful in discriminating the college student with and without
ADHD and can also be used as a screening tool.
There are several other clinical implications that could be considered when using the
BDEFS with the college student population. First, there were statistically significant differences
between the self-report and the other-informant report form of the BDEFS in this sample. This
result highlights the need for clinicians to gain not only the students’ perception of their
impairments, but the perceptions of those who interacted with the student the most. In addition,
104
results suggest that clinicians view the self-report rating with caution, taking into consideration
the student’s circumstances such as a crisis situation or the possibility of secondary gain.
Second, high cognitive ability students tend to have more impairment in the area of self-
management of time. Students in the category of high cognitive ability tend to be overlooked
because of their abilities; however, the results of this study highlight the need to provide a skills-
based intervention to improve skills for time-management. This may include setting alarms or
alerts for tasks and activities and encouraging the student with ADHD to have visual reminders
of time (countdown clocks on their desks).
105
APPENDIX A
INFORMED CONSENT FOR ADHD GROUP
Consent Form
1. I consent to receiving a psycho-educational assessment from the Adult Learning Evaluation Center at Florida State University.
2. I understand that no information concerning my evaluation will be released from the Adult Learning Evaluation Center within the limits of confidentiality that have been specified (see Client Information). 3. I understand the information provided to me regarding supervision and observation of services. 4. I understand that the fee for a psycho-educational assessment is $500.00 and is payable on the first day of the evaluation unless other arrangements have been finalized through financial aid. 5. I understand that it is in my best interest to put forth my best effort during the psycho-educational evaluation. 6. The following section specifically applies to a research project that you are being asked to consider.
I freely and voluntarily and without element of force or coercion, consent to be a participant in the research project, Exploration of the Factors Underlying Academic Difficulty in College
Students.
I understand that this research is being conducted by Dr. Frances Prevatt at Florida State University. I understand the purpose of the research project is to create an archival data base that can be used to evaluate correlates of learning disability (LD) and Attention Deficit Hyperactivity Disorder (ADHD) in a college population. I am being asked to allow the results of my current evaluation to be utilized in this archival data base. I understand that all clients in ALEC, (approximately 200 per year) are asked to participate in this research. I am not being asked to do
106
anything other than my standard evaluation; I am just allowing my data to be used later for research purposes. I understand that I must be at least 18 years of age in order to participate in this study. I understand that I will receive no direct benefits in return for participating in this research project. I understand that my participation is totally voluntary and I may withdraw my consent at any time in the research. I understand that if I do not agree for my data to be used, that will have no impact on my evaluation. I understand there is no risk involved if I agree to let my data be used. I understand that my identity will never be associated with the data (that is, my name and any identifying information will be removed.)The records will be kept private and confidential to the extent permitted by law. Data will be stored securely and only the researchers will have access to the data base. I understand that I may contact Dr. Frances Prevatt, Florida State University, Adult Learning Evaluation Center, at *********, for answers to questions about this research or my rights.
If you have any questions or concerns regarding this study and would like to talk to someone other than the researcher(s), you are encouraged to contact the FSU IRB at 2010 Levy Street, Research Building B, Suite 276, Tallahassee, FL 32306-2742, or 850-644-8633, or by email at [email protected]. I do [ ] do not [ ] consent to allow my data to be used in the manner described above. I do [ ] do not [ ] give ALEC my permission to contact me by email or telephone to describe future research projects and ask me if I would be interested in participating. If yes, this permission is granted for ____ years from today’s date. I do [ ] do not [ ] consent to participate in an additional research study that involves the comparison of my responses to those of a group of college students without ADHD. Should I agree, I will be given an additional thirty-three questions, which will add approximately ten minutes to my psycho-educational evaluation. I have read, understand, and agree to all Adult Learning Evaluation Center procedures outlined in this document. Signature ___________________________ Date__________________________
107
APPENDIX B
BARKLEY DEFICITS IN EXECUTIVE FUNCTIONING SCALES
Instructions: How often do you experience each of these problems? Please circle the number next to each item that best describes your behavior DURING THE PAST 6 MONTHS. Note to committee- these items are in an on-line survey tool, so the formatting here is not the
same. This is for a reference on the actual questions that are contained within the survey
All items are on a Likert scale with 1= Never or Rarely, 2= Sometimes, 3= Often, and 4= Very Often 1) Procrastinate or put off doing things until the last minute
2) Poor sense of time
3) Waste or mismanage my time
4) Not prepared on time for work or assigned tasks
5) Fail to meet deadlines for assignments
6) Have trouble planning ahead or preparing for upcoming events.
7) Forget to do things I am supposed to do
8) Can't seem to accomplish the goals I set for myself
9) Late for work or scheduled appointments
10) Can't seem to hold in mind things I need to remember to do
11) Can't seem to get things done unless there is an immediate deadline
12) Have difficulty judging how much time it will take to do something or get somewhere
13) Have trouble motivating myself to start work
14) Have difficulty motivating myself to stick with my work and get it done
15) Not motivated to prepare in advance for things I know I am supposed to do
16) Have trouble completing one activity before starting into a new one
17) Have trouble doing what I tell myself to do
18) Difficulties following through on promises or commitments I may make to others
19) Lack self-discipline
20) Have difficulty arranging or doing my work by its priority or importance; can't "prioritize"
well
21) Find it hard to get started or get going on things I need to get done
22) I do not seem to anticipate the future as much or as well as others
23) Can't seem to remember what I previously heard or read about
108
24) I have trouble organizing my thoughts
25) When I am shown something complicated to do, I cannot keep the information in mind so as
to imitate or do it correctly
26) I have trouble considering various options for doing things and weighing their consequences
27) Have difficulties saying what I want to say
28) Unable to come up with or invent as many solutions to problems as others seem to do
29) Find myself at a loss for words when I want to explain something to others
30) Have trouble putting my thoughts down in writing as well or as quickly as others
31) Feel I am not as creative or inventive as others of my level of intelligence
32) In trying to accomplish goals or assignments, I find I am not able to think of as many ways
of doing things as others
33) Have trouble learning new or complex activities as well as others
34) Have difficulty explaining things in their proper order or sequence
35) Can't seem to get to the point of my explanations as quickly as others
36) Have trouble doing things in their proper order or sequence
37) Unable to "think on my feet" or respond as effectively as others to unexpected events
38) I am slower than others at solving problems I encounter in my daily life
39) Easily distracted by irrelevant events or thoughts when I must concentrate on something
40) Not able to comprehend what I read as well as I should be able to do; have to reread material
to get its meaning
41) Cannot focus my attention on tasks or work as well as others
42) Easily confused
43) Can't seem to sustain my concentration on reading, paperwork, lectures, or work
44) Find it hard to focus on what is important from what is not important when I do things
45) I don't seem to process information as quickly or as accurately as others
46) Find it difficult to tolerate waiting; impatient
47) Make decisions impulsively
48) Unable to inhibit my reactions or responses to events or others
49) Have difficulty stopping my activities or behavior when I should do so.
50) Have difficulty changing my behavior when I am given feedback about my mistakes.
51) Make impulsive comments to others.
109
52) Likely to do things without considering the consequences for doing them.
53) Change my plans at the last minute on a whim or last minute impulse.
54) Fail to consider past relevant events or past personal experiences before responding to
situations (I act without thinking).
55) Not aware of things I say or do.
56) Have difficulty being objective about things that affect me.
57) Find it hard to take other people's perspectives about a problem or situation.
58) Don't think or talk things over with myself before doing something.
59) Trouble following the rules in a situation.
60) More likely to drive a motor vehicle much faster than others (Excessive speeding).
61) Have a low tolerance for frustrating situations
62) Cannot inhibit my emotions as well as others.
63) I don't look ahead and think about what the future outcomes will be before I do something (I
don't use my foresight).
64) I engage in risk taking activities more than others are likely to do.
65) Likely to take short cuts in my work and not do all that I am supposed to do.
66) Likely to skip out on work early if my work is boring to do.
67) Do not put as much effort into my work as I should or than others are able to do.
68) Others tell me that I am lazy or unmotivated.
69) Have to depend on others to help me get my work done.
70) Things must have an immediate payoff for me or I do not seem to get them done.
71) Have difficulty resisting the urge to do something fun or more interesting when I
am supposed to be working.
72) Inconsistent in the quality or quantity of my work performance.
73) Unable to work as well as others without supervision or frequent instruction.
74) I do not have the willpower or determination that others seem to have.
75) I am not able to work toward longer term or delayed rewards as well as others.
76) I cannot resist doing things that produce immediate rewards, even if those things are not
good for me in the long run.
77) Quick to get angry or become upset.
78) Overreact emotionally.
110
79) Easily excitable.
80) Unable to inhibit showing strong negative or positive emotions.
81) Have trouble calming myself down once I am emotionally upset.
82) Cannot seem to regain emotional control and become more reasonable once I am emotional.
83) Cannot seem to distract myself away from whatever is upsetting me emotionally to help
calm me down. I can't refocus my mind to a more positive framework.
84) Unable to manage my emotions in order to accomplish my goals successfully or get along
well with others.
85) I remain emotional or upset longer than others.
86) I find it difficult to walk away from emotionally upsetting encounters with others or leave
situations in which I have become very emotional.
87) I cannot re-channel or redirect my emotions into more positive ways or outlets when I get
upset.
88) I am not able to evaluate an emotionally upsetting event more objectively.
89) I cannot redefine negative events into more positive viewpoints when I feel strong emotions.
111
APPENDIX C
INFORMED CONSENT FOR CONTROL GROUP
Measure of Attention Deficit Hyperactivity Disorder (ADHD) and Executive Functioning in
College students
I understand that I am not required to take this survey and that I have the option to decline participation. If I agree, my responses will be used in the research project described below. I understand that I will take this survey online at the above stated web address. I will keep this copy of the informed consent for my records, but I will sign a copy of this consent online prior to completing the survey. I understand that this survey is being collected to serve a research study, entitled “The Psychometric Properties of the Barkley Deficits in Executive Functioning” You will be part of the control group without ADHD. If you have a current diagnosis of ADHD and you report that on the survey, then your data will not be used as part of the control group. However, you may still participate in the study and still be eligible to participate in the lottery. This study is being conducted by Dr. Frances Prevatt at Florida State University. I understand the purpose of this research project is to evaluate an existing measure that currently has no normative data for college students. I will be given a questionnaire to complete, which will take approximately 15 minutes. About 300 college students will participate in this study, 150 with a diagnosis of ADHD and 150 without a diagnosis of ADHD. I understand that I must be at least 18 years of age in order to participate in this study. I understand that in return for participating in this research project, I will be entered in a drawing for a 1 in 25 chance of receiving a $15 gift certificate to the store of my choosing. I understand that my participation is totally voluntary and I may stop participation at any time in the research, and that there is no penalty for non-participation. I understand this consent may be withdrawn at any time, even after I have completed the survey. I understand that the responses I provide today are being collected with software that is designed to secure my data and provide me with confidentiality. Although every effort will be done to ensure confidentiality of my responses, all Internet-based communication is subject to the remote likelihood of tampering from an outside source. IP addresses will not be investigated and data will be removed from the server. My data and consent form will be kept electronically on secure servers at the FSU Learning Systems Institute (LSI) and will not be disclosed to third parties. LSI has physical and environmental controls in place to protect data. Data are backed up daily.
112
I understand that I may contact the primary researcher, Dr. Frances Prevatt at *******. I can also contact the Chair of the Human Subjects Committee, Institutional Review Board, through the Office of the Vice President for Research, at ********. I freely and voluntarily and without element of force or coercion, consent to be a participant in the research project “The Psychometric Properties of the Barkley Deficits in Executive Functioning Scale.” It is possible that I may wonder about my responses to the questions. If after having answered the survey questions, I feel I may have some symptoms of ADHD, I can contact my local chapter for Children and Adults with Attention-Deficit/Hyperactivity Disorder (CHADD) at www.chadd.org for information for assistance with resources or I may contact the following resources: The FSU Student Counseling Center ****** (free) The FSU Psychology Department Clinic ********(sliding scale fee) The FSU Human Services Center ******* (free) If you have any questions or concerns regarding this study and would like to talk to someone other than the researcher(s), you are encouraged to contact the FSU IRB at 2010 Levy Street, Research Building B, Suite 276, Tallahassee, FL 32306-2742, or 850-644-8633, or by email at [email protected] You will be given a copy of this information to keep for your records. Statement of Consent:
I have read the above information. I have asked questions and have received answers. I consent to participate in the study. _____ YES. By checking yes, I consent to participate in this study. ________________ _________________ Signature Date ________________ _________________ Signature of Investigator Date
113
APPENDIX D
INTERNAL REVIEW BOARD FOR HUMAN SUBJECTS APPROVAL
The Florida State University Office of the Vice President For Research Human Subjects Committee Tallahassee, Florida 32306-2742 (850) 644-8673, FAX (850) 644-4392 APPROVAL MEMORANDUM Date: 3/29/2013 To: Frances Prevatt Dept.: EDUCATIONAL PSYCHOLOGY AND LEARNING SYSTEMS From: Thomas L. Jacobson, Chair Re: Use of Human Subjects in Research The Psychometric Properties of the Barkley Deficits in Executive Functioning Scale (BDEFS) The application that you submitted to this office in regard to the use of human subjects in the proposal referenced above have been reviewed by the Secretary, the Chair, and one member of the Human Subjects Committee. Your project is determined to be Expedited per 45 CFR § 46.110(7) and has been approved by an expedited review process. The Human Subjects Committee has not evaluated your proposal for scientific merit, except to weigh the risk to the human participants and the aspects of the proposal related to potential risk and benefit. This approval does not replace any departmental or other approvals, which may be required. If you submitted a proposed consent form with your application, the approved stamped consent form is attached to this approval notice. Only the stamped version of the consent form may be used in recruiting research subjects. If the project has not been completed by 3/28/2014 you must request a renewal of approval for continuation of the project. As a courtesy, a renewal notice will be sent to you prior to your expiration date; however, it is your responsibility as the Principal Investigator to timely request renewal of your approval from the Committee. You are advised that any change in protocol for this project must be reviewed and approved by the Committee prior to implementation of the proposed change in the protocol. A protocol change/amendment form is required to be submitted for approval by the Committee. In addition, federal regulations require that the Principal Investigator promptly report, in writing any unanticipated problems or adverse events involving risks to research subjects or others.
114
By copy of this memorandum, the Chair of your department and/or your major professor is reminded that he/she is responsible for being informed concerning research projects involving human subjects in the department, and should review protocols as often as needed to insure that the project is being conducted in compliance with our institution and with DHHS regulations. This institution has an Assurance on file with the Office for Human Research Protection. The Assurance Number is FWA00000168/IRB number IRB00000446. Cc: Betsy Becker, Chair HSC No. 2013.10087 Human Subjects Application For Full IRB and Expedited Exempt Review
1. Project Title and Identification
1.1 Project Title
The Psychometric Properties of the Barkley Deficits in Executive Functioning Scale (BDEFS)
Project is: Dissertation
1.2 Principal Investigator (PI)
Name(Last name, First name MI): Prevatt, Frances F
Highest Earned
Degree: Doctorate
University Department: EDUCATIONAL PSYCHOLOGY AND LEARNING SYSTEMS
Email:
The training and education completed in the protection of human
subjects or human subjects records: Other
Occupational
Position: Faculty
1.3 Co-Investigators/Research Staff
Name(Last name, First name MI): Coffman, Theodora Passinos; Co-Investigator
Highest Earned
Degree: Bachelor's
Degree
University Department: EDUCATIONAL PSYCHOLOGY AND LEARNING SYSTEMS
Email:
The training and education completed in the protection of human subjects
or human subjects records: FSU Training Module
Occupational
Position: Student
115
APPENDIX E
DEMOGRAPHIC QUESTIONNAIRE
If you consent to taking this survey, please select yes with the knowledge that your information will be kept confidential and used for research purposes only.
Yes No
If No Is Selected, Then Skip To End of Survey
1) What is your gender?
Male Female
2) What is your age?
_________________ (in years) 3) What is your ethnicity?
Caucasian African American Asian Hispanic Other
4) What year in college are you in?
Freshmen Sophomore Junior Senior Graduate Student
5) Have you been previously diagnosed with a Learning Disability?
Yes No
6) Have you been previously diagnosed with ADHD?
Yes No
116
APPENDIX F
BDEFS SCORING TEMPLATE
Scale Raw Score Percentile rank
Classification
Section 1, Q-1-21 Self-Management/ To Time
Section 2, Q-22-45 Self-Organization/ Problem Solving
Section 3, Q-46-64 Self-Restraint
Section 4, Q-65-76 Self-Motivation
Section 5, Q-77-89 Self-Regulation of Emotions
Total sections 1-5 Total EF
Count # of items answered 3 or 4
EF symptom count
Add items, 1,6,14,16,24,49,50,55,60 65, 69
ADHD-EF index score
117
APPENDIX G
OTHER INFORMANT BDEFS
Note to committee members- the questions contained in the other-informant version of the
BDEFS are identical in content to the self-report version. The only change is in first/third
person reference.
1) Procrastinates or puts off doing things until the last minute
2) Poor sense of time
3) Waste or mismanage his/her time
4) Not prepared on time for work or assigned tasks
5) Fails to meet deadlines for assignments
6) Has trouble planning ahead or preparing for upcoming events.
7) Forgets to do things that he/she am supposed to do
8) Can't seem to accomplish the goals he/she set for self
9) Late for work or scheduled appointments
10) Can't seem to hold in mind things he/she need to remember to do
11) Can't seem to get things done unless there is an immediate deadline
12) Has difficulty judging how much time it will take to do something or get somewhere
13) Has trouble motivating self to start work
14) Has difficulty motivating self to stick with his/her work and get it done
15) Not motivated to prepare in advance for things he/she knows he/she is supposed to do
16) Has trouble completing one activity before starting into a new one
17) Has trouble doing what he/she tells self to do
18) Difficulties following through on promises or commitments he/she may make to others
19) Lack self-discipline
20) Has difficulty arranging or doing his/her work by its priority or importance; can't "prioritize"
well
21) Finds it hard to get started or get going on things he/she need to get done
22) Does not seem to anticipate the future as much or as well as others
23) Can't seem to remember what he/she previously heard or read about
24) Has trouble organizing his/her thoughts
118
25) When he/she is shown something complicated to do, he/she cannot keep the information in
mind so as to imitate or do it correctly
26) Has trouble considering various options for doing things and weighing their consequences
27) Has difficulties saying what he/she wants to say
28) Unable to come up with or invent as many solutions to problems as others seem to do
29) Finds he/she is at a loss for words when he/she wants to explain something to others
30) Has trouble putting his/her thoughts down in writing as well or as quickly as others
31) Feels he/she is not as creative or inventive as others of his/her level of intelligence
32) In trying to accomplish goals or assignments, he/she finds that he/she is not able to think of
as many ways of doing things as others
33) Has trouble learning new or complex activities as well as others
34) Has difficulty explaining things in their proper order or sequence
35) Can't seem to get to the point of his/her explanations as quickly as others
36) Has trouble doing things in their proper order or sequence
37) Unable to "think on his/her feet" or respond as effectively as others to unexpected events
38) Is slower than others at solving problems he/she encounters in his/her daily life
39) Easily distracted by irrelevant events or thoughts when he/she must concentrate on
something
40) Not able to comprehend what he/she read as well as he/she should be able to do; has to
reread material to get its meaning
41) Cannot focus his/her attention on tasks or work as well as others
42) Easily confused
43) Can't seem to sustain his/her concentration on reading, paperwork, lectures, or work
44) Finds it hard to focus on what is important from what is not important when he/she does
things
45) Doesn’t seem to process information as quickly or as accurately as others
46) Finds it difficult to tolerate waiting; impatient
47) Makes decisions impulsively
48) Unable to inhibit his/her reactions or responses to events or others
49) Has difficulty stopping his/her activities or behavior when he/she should do so.
50) Has difficulty changing his/her behavior when he/she is given feedback about my mistakes.
119
51) Makes impulsive comments to others.
52) Likely to do things without considering the consequences for doing them.
53) Changes his/her plans at the last minute on a whim or last minute impulse.
54) Fails to consider past relevant events or past personal experiences before responding to
situations (Acts without thinking).
55) Not aware of things he/she says or does.
56) Has difficulty being objective about things that affect him/her.
57) Finds it hard to take other people's perspectives about a problem or situation.
58) Doesn’t think or talk things over with self before doing something.
59) Trouble following the rules in a situation.
60) More likely to drive a motor vehicle much faster than others (Excessive speeding).
61) Has a low tolerance for frustrating situations
62) Cannot inhibit his/her emotions as well as others.
63) Doesn’t look ahead and think about what the future outcomes will be before he/she does
something (Doesn’t use his/her foresight).
64) Engages in risk taking activities more than others are likely to do.
65) Likely to take short cuts in his/her work and not do all that he/she is supposed to do.
66) Likely to skip out on work early if his/her work is boring to do.
67) Does not put as much effort into his/her work as he/she should or than others are able to do.
68) Others tell his/her that he/she is lazy or unmotivated.
69) Has to depend on others to help them get their work done.
70) Things must have an immediate payoff for his/her or he/she does not seem to get them done.
71) Has difficulty resisting the urge to do something fun or more interesting when he/she is
supposed to be working.
72) Inconsistent in the quality or quantity of his/her work performance.
73) Unable to work as well as others without supervision or frequent instruction.
74) Does not have the willpower or determination that others seem to have.
75) Is not able to work toward longer term or delayed rewards as well as others.
76) Cannot resist doing things that produce immediate rewards, even if those things are not good
for him/her in the long run.
77) Quick to get angry or become upset.
120
78) Overreact emotionally.
79) Easily excitable.
80) Unable to inhibit showing strong negative or positive emotions.
81) Has trouble calming self down once he/she is emotionally upset.
82) Cannot seem to regain emotional control and become more reasonable once he/she is
emotional.
83) Cannot seem to distract self away from whatever is upsetting him/her emotionally to help
calm self down. Can't refocus his/her mind to a more positive framework.
84) Unable to manage his/her emotions in order to accomplish his/her goals successfully or get
along well with others.
85) Remains emotional or upset longer than others.
86) Find it difficult to walk away from emotionally upsetting encounters with others or leave
situations in which he/she has become very emotional.
87) Cannot re-channel or redirect his/her emotions into more positive ways or outlets when
he/she gets upset.
88) Is not able to evaluate an emotionally upsetting event more objectively.
89) Cannot redefine negative events into more positive viewpoints when he/she feels strong
emotions.
121
REFERENCES
Alderman, N., Burgess, P. W., Kight, C., & Henman, C. (2003). Ecological validity of a simplified version of the multiple errands shipping test. Journal of the International
Neuropsychological Society, 9, 31-44 The American Educational Research Association, the American Psychological Association, and
the National Council on Measurement in Education (1999). Standards for education and
psychological testing. Washington D.C.: American Educational Research Association.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental
disorders-Text Revision (4th ed.). Washington, DC: Author. Anderson, P. (2002). Assessment and development of executive function (EF) during childhood.
Neuropsychology, 8, 71-82. doi: 0929-7049/02/0802-071 Anderson, V., Anderson, P., Northman, E., J., & Mikiewicz, O. (2002). Relationships between
cognitive and behavioral measures of executive functioning in children with brain disease. Child Neuropsychology, 8(4), 231-240. doi: 0929-7049/02/0804-231
Antshel, K. M., Faraone, S. V., Maglione, K., Doyle, A. E., Seidman, L. J., & Biderman, J.
(2010). Executive functioning in high-IQ adults with ADHD. Psychological Medicine,
40, 1909-1918. doi:10.1017/S0033291709992273 Antshel, K. M., Faraone, S. V., Maglione, K….Biederman, J. (2009). Is adult attention deficit
hyperactivity disorder a valid diagnosis in the presence of high IQ? Psycholgical
Medcine, 39(8), 1325-1335. doi: 10.1017/S0033291708004959 Ardila, A., Ostrosky-Solis, F., Rosselli, M., & Gomez, C. (2000). Age related cognitive decline
during normal aging: The complex effect of education. Archives of Clinical
Neuropsychology, 15, 495–514. Barkley, R. A. (1997a). ADHD and the nature of self-control. New York: Guilford Press. Barkley, R. A. (1997b). Inhibition, sustained attention, and executive functions: constructing a
unifying theory of ADHD. Psychological Bulletin, 121, 65–94. N doi: 10.1037/0033-2909.121.1.65
Barkley, R. A. (2011a). Barkley Adult ADHD Rating Scale (BAARS-IV): The Guilford Press. Barkley, R. A. (2011b). Barkley Deficits in Executive Functioning Scale (BDEFS). New York:
Guilford.
122
Barkley, R. A. (2011c). Executive functioning and self-regulation in Adults with ADHD: Nature, Assessment and Treatment. Symposium at Children and Adults with ADHD (CHADD) Conference, Orlando, FL.
Barkley, R. A. (2012). Executive functions: What they are, how they work, and why they evolved.
New York: Guilford Press. Barkley, R. A., & Fischer, M. (2010). The unique contribution of emotional impulsiveness to
impairments in major life activities in hyperactive children as adults. Journal of the
American Academy of Child and Adolescent Psychiatry, 49(5), 503-513. Barkley, R.A., & Murphy, K. R. (2010). Impairment in occupational functioning and adult
ADHD: The predictive utility of executive function (EF) ratings versus EF tests. Archives
of Clinical Neuropsychology, 25, 157–173. Barkley, R. A., Murphy, K. R., & Fischer, M. (2008). ADHD in adults: What the science says.
New York: Guildford Press. Baxter, M. G. & Murray, E. A. (2002). The amygdala and reward. National Review of
Neuroscience, 3, 563-573. Biederman, J., & Faraone, S. V. (2006). The effects of Attention-Deficit/Hyperactivity Disorder
on employment and household income. Medscape General Medicine, 8, 1-6. Biederman, J., Faraone, S.V., Spencer, T.J., Mick, E., Monuteaux, M.C., Aleardi, M. (2006).
Functional impairments in adults with self-reports of diagnosed ADHD: a controlled study of 1001 adults in the community. Journal of Clinical Psychiatry, 67, 524–540.
Biederman, J., Faraone, S. V., Spencer, T., Wilens, T., Norman, D., Lapey, K. A., …Doyle, A.
(1993). Patterns of psychiatric comorbidity, cognition, and psychosocial functioning in adults with attention deficit hyperactivity disorder. American Journal of Psychiatry 150, 1792–1798.
Biederman, J., Mick, E., Faraone, S.V. (2000). Age-dependent decline of symptoms of attention
deficit hyperactivity disorder: impact of remission definition and symptom type. American Journal of Psychiatry, 157, 816–818.
Biederman, J., Petty, C. R., Fried, R., Doyle, A. E., Spencerm T….. Seidman, L., (2007).
Stability of executive function deficits into young adult years: A prospective longitudinal follow-up study of grown up males with ADHD. Acta Psychiatrica Scandinavia, 116, 129-136.
Biederman, J., Spencer, T., & Wilens, T. (2004). Evidence-based pharmacotherapy for
attention-deficit hyperactivity disorder. International Journal of
Neuropsychopharmacology, 7, 77-97.
123
Biederman, J., Spencer, T., Wilens, T., Prince, J. B., & Faraone, S. V. (2006). Treatment of ADHD with stimulant medications: Response to Nissen perspective in The New England Journal of Medicine. Journal of the American Academy of Child and Adolescent
Psychiatric, 45, 10, 1147-1150. Booksh, R. L., Pella, R. D., Singh, A. N., and Gouvier, W. D. (2010). Ability of college students
to simulate ADHD on objective measures of attention. Journal of Attention Disorder, 13, 325-338.
Boonstra, A. M., Oosterlaan, J., Sergeant, J. A., & Buitelaar, J. K. (2005). Executive functioning
in adult ADHD: A meta-analytic review. Psychological Medicine, 35, 1097–1108. Brown, T. A. (2006). Confirmatory factor analysis for applied research. Sage Publication: New
York, NY. Brown, T. E. (2008). ADD/ADHD and impaired executive function in clinical practice. Current
Psychiatry Reports (10), 51 407-411. Brown, B.M. and Cutik, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S.
Long (Eds). Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
Burgess, P. W., Alderman, N., Evans, J., Emsile, H., & Wilson, B. A. (1998.). The ecological validity of tests of executive functioning. Journal of the International
Neuropsychological Society, 4, 457-558. Burgess, P. W., & Shallice, T. (1996). Response suppression, initiation and strategy use
following frontal lobe lesions. Neuropsychologia, 34, 263–272. Burgess, P. W., & Shallice, T. (1997). The Hayling and Brixton Tests. Bury St. Edmunds (UK):
Thames Valley Test Company. Carmines, E. & Zeller, R. (1979). Reliability and validity assessment. Thousand Oaks, CA: Sage
Publications, Inc.
Castellanos, F.X., Sonuga-Barke, E.J.S., Milham, M.P, & Tannock, R. (2006). Characterizing cognition in ADHD: Beyond executive dysfunction. Trends in Cognitive Sciences, 10,
117-123. doi: 10.1016/j.tics.2006.01.011 Chaytor, N. S (2004). Improving the ecological validity of executive functioning assessment.
(Unpublished doctoral dissertation). Washington State University, Washington. Chaytor, N., Schmitter-Edgecombe, M., & Burr, R. (2006). Improving the ecological validity of
executive functioning assessment. Archives of Clinical Neuropsychology, 21, 217-227. doi:10.1016/j.acn.2005.12.002
124
Chelune, G. J., Ferguson, W., Koon, R., & Dickey, T. O. (1996). Frontal lobe disinhibition in attention deficit disorder. Child Psychiatry and Human Development, 16, 221-234.
Clark, C., Prior, M., & Kinsella, G. (2000). Do executive function deficits differentiate between
adolescents with ADHD and oppositional defiant/conduct disorder? A neuropsychological study using the six elements test and Hayling sentence completion test. Journal of Abnormal Child Psychology, 28(5), 403-414.
Comrey, A.L., & Lee, H.B. (1992). A first course in factor analysis. (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. Conners, C. K. (1995). Conners’ Continuous Performance Test. Toronto: Multi-Health Systems. Cortese, S., Kelly, C., Chabernaud, C., Proal, E., Di Martino, A., Milham, M. P., & Castellanos,
F. Z. (2012). Towards systems neuroscience of ADHD: A meta-analysis if 55 fMRI studies. American Journal of Psychiatry, 169, 1038-1055.
Dawkins, R. (1982). The extended phenotype: The long reach of the gene. New York: Oxford
University Press. Delis, D.C., Kaplan, E., & Kramer, J.H. (2001). The Delis Kaplan Executive Function System:
Examiner’s Manual. San Antonio: The Psychological Corporation. Denckla, M. B. (1996). A theory and model of executive function: A neuropsychological
perspective. In G. Lyon, & N. Krasnegor (Eds.), Attention, memory and executive
function (pp 263-278). Maryland: Paul Brooks. Doehnert, M., Brandeis, D., Schneider, G., Drechsler, R., & Steinhausen, H. C. (2012). A
neurophysiological marker of impaired preparation in an 11-year follow –up study of attention-deficit/hyperactivity disorder (ADHD). The Journal of Child Psychology and
Psychiatry. doi: 10.1111/j.1469-7610.2012.02572.x. Duncan, J., Emslie, H., Williams, P., Johnson, R., & Freer, C. (1996). Intelligence and the frontal
lobes: The organization of goal-directed behavior. Cognitive Psychology, 30, 257–303. DuPaul, G. J., Schaughency, E. A., Weyandt, L. I., Tripp, G., Kiesner, J., Ota, K., et al. (2001).
Self-report of ADHD symptoms in university students: Gender and cross-national prevalence. Journal of Learning Disabilities, 34, 370-379.
DuPaul, G. J., Weyandt, L. L., O’ Dell, S. M., & Varejao, M. (2009). College students with
ADHD: Current status and future directions. Journal of Attention Disorder, 13, 234-250. Eslinger, P. J. (1996). Conceptualizing, describing, and measuring components of executive
function: A summary. In G. R. Lyon & N. A. Krasnegor (Eds.), Attention, memory, and
executive function (pp. 367-395). Baltimore: Brookes.
125
Faraone, S. V., Wilens, T.E., Petty, C., Antshel, K., Spencer, T., Biederman, J. (2007). Substance use among ADHD adults: Implications of late onset and subthreshold diagnoses. American Journal on Addictions 16 (Suppl. 1), 24–32, quiz 33–34.
Faraone, S. V., Biederman, J., Spencer, T., Mick, E., Murray, K., Petty, C…Monuteaux, M. C.
(2006). Diagnosing adult attention deficit hyperactivity disorder: Are late onset and sub-threshold diagnoses valid? American Journal of Psychiatry, 163, 1720-1729.
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.G. (2013). Statistical power analysis using
G*Power 3.1.6: Tests for correlation and regression analyses. Behavior Research
Methods, 41, 1149-1160. Fennerty, D., Lambert, C., Majsterek, D. (2000). Behavior Rating Scales: An Analysis. Eric
Document #ED442042. Fink, A., & Litwin, M. (1995). How to measure survey reliability and validity. Sage Publications
Inc.: Thousand Oaks, California. Fischer, M., Newby, R., & Gordon, M. (1995). Who are the false negatives on continuous
performance tests? Journal of Clinical Child Psychology, 24(4), 427–433. Floyd, F. J. & Widaman, K. F. (1995). Factor Analysis in the development and refinement of
clinical assessment instruments. Psychological Assessment, 7, 286-299. Frazier, T. W., Youngstrom, E. A., Glutting, J. J., & Watkins, M. W. (2007). ADHD and
achievement: Meta-analysis of the child, adolescent, and adult literatures and a concomitant study with college students. Journal of Learning Disabilities, 40(1), 49-65.
Forbes, G. B. (1998). Clinical utility of the test of variables of attention (TOVA) in the diagnosis
of attention deficit/hyperactivity disorder. Journal of clinical psychology, 54(4), 461-476.
Fuster, J.M. (1997). The prefrontal cortex-anatomy physiology, and neuropsychology of the
frontal lobe (3rd ed.). Philadelphia: Lippincott-Raven. Gioia, G. A., Isquith, P. K., Guys, S. C., & Kenworthy, L. (2000). BREIF: Behavior Rating
Inventory of Executive Functions- Professional Manual. Odessa, FL: Psychological Assessment Resources.
Glutting, J. J., Youngstrom, E. A., & Watkins, M. W. (2005). ADHD and college students:
Exploratory and confirmatory factor structures with student and parent data. Psychological Assessment, 17, 44-55.
Gonon, F. (2008). The dopaminergic hypothesis of attention-deficit/hyperactivity disorder needs
re-examining. Trends in Neurosciences, 32(1), 1-8. doi:10.1016/j.tins.2008.09.010
126
Goodwin, J. (2010). Research in psychology: methods and design. John Wiley & Sons: Hoboken, NJ.
Gordon, M. (1983). The Gordon Diagnostic System. DeWitt, NY: Gordon Systems. Gordon, M., & Keiser, S. (1998). Accommodations in higher education under the Americans
With Disabilities Act (ADA): A no-nonsense guide for clinicians, educators,
administrators, and lawyers. DeWitt, NY: GSI Publications. Grant, D., & Berg, E. A. (1948). Wisconsin Card Sorting Test. Los Angeles: Western Psychological Corporation.
Graham, J.W. (2009). Missing data analysis: Making it work in the real world. Annual Review of
Psychology, 60, 549-576. doi: 10.1146/annurev.psych.58.110405.085530 Greenberg, L. M. (1991). TOVA tm interpretation manual. Minneapolis, MN: Grenwald-Mayes, G. (2002). Relationship between current quality of life and family of origin
dynamics for college students with attention-Deficit/Hyperactivity disorder. Journal of
Attention Disorders, 5, 211-222. Grondzinsky, G. M., & Diamond, R. (1992). Frontal lobe functioning in boys with attention-
deficit/hyperactivity disorder. Developmental Neuropsychology, 8, 427-445. Gulliksen, H. (1950). Intrinsic validity. American Psychologist, 5, 511-517. Heaton, R. (1981). Wisconsin card sorting test: Manual. Odessa: Psychological Assessment
Resources, Inc. Heiligenstein, E., Guenther, G., Levy, A., Savino, F., & Fulwiler, J. (1999). Psychological and
academic functioning in college students with attention deficit hyperactivity disorder. Journal of American College Health, 47, 181-185.
Heiman, G. W. (2002). Research methods in Psychology. Boston, MA: Houghton Mifflin Co. Hervey, A. S., Epstein, J. N., & Curry, J. F. (2004). Neuropsychology of adults with attention-
deficit/hyperactivity disorder: a meta-analytic review. Neuropsychology, 18, 495-503. Hoffman, W., Friese, M., Schmeichel, B. J., & Baddeley, A. D. (2011). Working memory and
self-regulation. In K. D. Vohs & R. F. Baumeister (Eds.), Handbook of self-regulation:
Research, theory, and applications (2nd ed., pp. 41-63). New York: Guilford Press. Homack, S., Lee, D., & Riccio, C. A. (2005). Test review: Delis-Kaplan Executive Function
System. Journal of Clinical and Experimental Neuropsychology, 27, 599-609. Hood, A. B. & Johnson, R. W. (2006). Assessment in counseling: A guide to the use of
psychological assessment procedures (4th ed.). American Counseling Association: Alexandra, VA.
127
Hu, T., and Bender, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria verses new alternatives. Structure Equation Modeling, 6, 1-55.
Jackson, S. L. (2008). Research methods: a modular approach, Thomson Wadsworth: Belmont,
CA. Jung, R. E., Yeo, R. A., Chiulli, S. J., Sibbitt, W. L., & Brooks, W. M. (2000). Myth of
neuropsychology: Intelligence, neurometabolism, and cognitive ability. The Clinical
Neuropsychologist 14(4), 535-545. Kassubek, J., Juengling, F. D., Ecker, D., & Landwehrmeyer, G. B. (2005). Thalamic atrophy in
Huntington’s disease co-varies with cognitive performance: A morphometric MRI analysis. Cerebral Cortex, 15, 846–853.
Kessler, R. C., Heeringa, S., Lakoma, M. D., Petukhova, M., Rupp, A., Schoenbaum,
M…Zaslavsky, A. M., (2010). Individual and societal effects of mental disorders on earnings in the United States: Results from the national comorbidity survey replication. The American Journal of Psychiatry, 165, 703-711. doi:10.1176/appi.ajp.2008.08010126
Koechlin, E., Corrado, G., Pietrini, P., & Grafman, J. (2000). Dissociating the role of the medial
and lateral anterior prefrontal cortex in human planning. Proceedings of the National
Academy of Sciences, 97(13), 7651-7656. Korkman, M., Kirk, U., & Kemp, S. (1998). NEPSY: A developmental neuropsychological
assessment. San Antonio, TX: The Psychological Corporation. Levin, H., & Hanten, G. (2005). Executive functions after traumatic brain injury in children.
Pediatric Neurology, 33, 79-93. doi:10.1016/j.pediatrneurol.2005.02.002 Lewis, S. J., Dove, A., Robbins, T. W., Baker, R. A., & Owen, A. M. (2004). Striatal
contributions to working memory: A functional magnetic resonance imaging study in humans. European Journal of Neuroscience, 19, 755-760.
Lewis, R. F., Kelland, D. Z., Kupke, T. (1990). A normative study of the Repeatable Cognitive-
Perceptual-Motor Battery. Archives of Clinical Neuropsychology, 5, 201-214. Lezak, M. D., Howieson, D. B., Loring, D. W. (2004). Neuropsychological Assessment (4th ed.).
New York: Oxford University Press. Loo, S. K., Hale, T. S., Masion, J., Hanada, G., McGough, J. J., McCracken, J. T., & Smalley, S.
L. (2009). Cortical activity patterns in ADHD during arousal, activation and sustained attention. Neuropsychological, 47(10), 2114-2119. Doi: 10.1016/j.neuropsychologicaq.2009.04.03
Lowry, R. VassarStats: Website for Statistical Computation. (2013). Retrieved April 6, 2013,
from http://vassarstats.net/index.html
128
Luria, A. R. (1966). Higher cortical functions in man. New York: Basic Books. Mahone, E. M., Pillion, J. P., Hoffman, J., Hiemenz, J. R., & Denckla, M. B. (2002). Construct
validity of the Auditory Continuous Performance Test for Preschoolers. Developmental
Neuropsychology, 27, 11-33. Matarazzo, J. D., Wiens, A. N., Matarazzo, R. G., & Goldstein, S. G. (1974). Psychometric and
clinical test-retest reliability of the Halstead Impairment Index in a sample of healthy, young, normal men. Journal of Nervous and Mental Disease, 158, 37–49.
McCabe, S. E., Knight, J., Teter, C. J., & Wechsler, H. (2005). Nonmedical use of prescription
stimulants among US college students: Prevalence and correlates from a national survey. Addiction, 99, 96–106.
McGee, R. A., Clark, S. E., & Symons, D. K., (2000). Does the Conners’ Continuous
Performance Test aid in ADHD diagnosis? Journal of Abnormal Child Psychology,
28(5), 415-424. Meaux, J. B., Green, A., & Broussard, L. (2009). ADHD in the college student: a block in the
road. Journal of Psychiatric and Mental Health Nursing, 16, 248-256. Merrell, K. W. (1994). Assessment of behavioral, social, & emotional problems. New York:
Longman. Messick, S. (1989). Meaning and value in test validation: The science and ethics of assessment.
Educational Researcher, 18, 5-11. Mises, L. (1990). Human action: A treatise on economics (3rd ed.). San Francisco: Laissez-Faire
Brooks. (Original work published 1948). Miyake, A., Friedman, N., Emerson, M., Witzki, A., & Howerter, A. (2000). The unity and
diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100. doi:10.1006/cogp.1999.0734
Monchi, O., Petrides, M., Strafella, A. P., Worsley, K. J., & Doyon, J. (2006). Functional role of
the basal ganglia in the planning and execution of actions. Annual Neurology, 59, 257-264.
Murphy, K. (2005). Psychosocial treatments for ADHD in teens and adults: A practice-friendly
review. Journal of Clinical Psychology, 61, 607–619. Murphy, K., Barkley, B., & Tracie, B. (2001). Executive functioning and olfactory in young
adults with attention deficit-hyperactivity disorder. Neuropsychology, 15(2), 211-220. doi: 10.1037/0894-4105.15.2.211
129
Myers, L. S., Gamst, G., & Guarino, A. J. (2006). Applied multivariate research: design and
interpretation. Sage Publications: Thousand Oaks, CA Niendam, T. A., Laird, A. R., Ray, K. L., Dean, M. Y., Glahn, D. C., & Carter, C. S. (2012). Meta-analytic evidence for a superordinate cognitive control network subserving diverse executive functions. Cognitive Affective Behavioral Neuroscience, 12, 241-268.
Nadeau, K. G. (2005). Career choices and workplace challenges for individuals with ADHD.
Journal of Clinical Psychology, 61, 549–563. Naglieri, J. A., & Goldstein, S. (2013). Comprehensive Executive Functioning Inventory. NY:
Multi-Health Systems Inc. Nigg, J. T., & Casey, B. J. (2005). An integrative theory of attention-deficit/hyperactivity
disorder based on the cognitive and affective neurosciences. Development and
Psychopathology, 17, 765-806. Norwalk, K., Norvilitis, J. M., & MacLean, M.G., (2008). ADHD symptomatology and its
relationship to factors associated with college adjustment. Journal of Attention Disorders, 1-8.
Oosterlaan, J., Scheres, A., & Sergeant, J. A. (2005). Which executive functioning deficits are
associated with AD/HD, ODD/CD and comorbid AD/HD+ODD/CD? Journal of
Abnormal Psychology, 33, 69-85. Prevatt, F., Li, H., Welles, T., Festa-Dreher, D., Yelland, S., and Lee., J. (2011). The
Academic Success Inventory for College Students: Scale development and practical implications for use with students. Journal of College Admission, 211,
26-31. Proctor, B. E., & Prevatt, F. (2009). Confirming the factor structure of attention-
deficit/hyperactivity disorder symptoms in college students using student and parent data. Journal of Learning Disabilities, 42, 250-259.
Ramirez, C. A., Rosen, L. A., Deffenbacher, J. L., Hurst, H., Nicoletta, C., Rosencranz, T. A., et
al. (1997). Anger and anger expression in adults with high ADHD symptoms. Journal of Attention Disorders, 2, 115-128.
Rapoport, J. L., Buchsbaum, M. S., Zahn, T. P., Ludlow, C., & Mikkelsen, E. J. (1978, February
3). Dextroamphetamine: Cognitive and behavioral effects in normal prepubertal boys. Science, 199, 560–563.
Reaser, A., Prevatt, F., Petscher, Y., & Proctor, B. (2007). The learning and study strategies of
college students with ADHD. Psychology in the Schools, 44, 627–638. Reis, H. T., & Judd, C. M. (Eds.). (2000). Handbook of research methods in social and
personality psychology. Cambridge University Press.
130
Reitan, R. M., & Wolfson, D. (1985). The Halstead–Reitan Neuropsycholgical Test Battery:
Therapy and clinical interpretation. Tucson, AZ: Neuropsychological Press.
Rey, A. (1941). L’examen psychologique dans les cas d’encephalopathie traumatique. Arch.
Psychol. 28, 286–340.
Roth, R. M., Isquith, P. K., & Gioia, G. A. (2005). Behavior Rating Inventory of Executive
Function- Adult Version. FL: Psychological Assessment Resources. Rottschy, C., Langer, R., Dogan, I., Reetz, K., Laird, A. R., Schulz, J. B…Eickhoff, S. B. (2012).
Modelling neural correlates of working memory: A coordinate-based meta-analysis. Neuroimage, 60, 830-846.
Rueda, M. R., Posner, M. L., & Rothbard, M. K. (2011). Attentional control and self-regulation.
In K. D. Vohs & R. F. Baumeister (Eds.), Handbook of self-regulation: Research, theory,
and applications (2nd ed., pp. 41-63). New York: Guilford Press. Salthouse, T. (1996). The processing-speed theory of adult age differences in cognition.
Psychological Review, 103, 403–428. Salthouse, T. (2005). Relations between cognitive abilities and measures of executive
functioning. Neuropsychology, 19, 532–545. Schmidt, M. (2003). Hit or miss? Insight into executive functions. Journal of the International
Neuropsychological Society, 9, 962–964. Seidman, L. J., Biederman, J., Valera, E. M., Monuteaux, M. C., Doyle, A. E., & Faraone, S. V.
(2006). Neuropsychological functioning in girls with attention-deficit/hyperactivity disorder with an without learning disabilities. Neuropsychology, 20, 166-177.
Shallice, T. & Evans, M. E. (1978). The involvement of the frontal lobes in cognitive estimation.
Cortex, 14, 294-303. Stavro, G. M., Ettenhofer, M. L., & Nigg, J. T. (2007). Executive functions and adaptive
functioning in young adults with attention-deficit/hyperactivity disorder. Journal of the
International Neuropsychological Society, 13, 324-334. Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental
Psychology, 18, 643–662. Stuss, D. T. & Alexander, M. P. (2000). Executive functions and the frontal lobes: a conceptual
view. Psychological Research, 63, 289-298.
131
Stuss, D. T., Alexander, M. P., Floden, D., Binns, M. A., Levin, B., & McIntosh, A. R. (2002). Fractionation and localization of distinct frontal lobe processes: Evidence from focal lesions in humans. In D. T. Stuss, & R. T. Knight (Eds.), Principles of frontal lobe
function (pp. 392-407). New York, NY: Oxford University Press. Thompson, J. M., Gray, J. M., Crawford, J. R., Hughes, J. H., Young, A. H., & Ferrier, I. N.
(2009). Differential deficit in executive control in euthymic bipolar disorder. Journal of
Abnormal Psychology, 118, 146-160. Trenerry, M., Crosson, B., Deboe, J., & Leber, W. (1989). Stroop Neuropsychological
Screening Test. Odessa, FL: Psychological Assessment Resources. Turnock, P. M., Rosen, L. A., & Kaminski, P. L. (1998). Academic coping in college students
self-reporting high and low symptoms of attention deficit hyperactivity disorder. Journal of College Student Development, 39, 484-493.
Tuttle, J., Scheurich, N., & Ranseen, J. D. (2007, May). Prevalence and attitudes concerning
ADHD diagnosis and use of stimulant medication in medical students. Paper presented at the annual meeting of the American Psychiatric Association, San Diego, CA.
Vriezen, E. R., & Pigott, S. E. (2002). The relationship between parental report on the BRIEF
and performance based measures of executive function in children with moderate to severe traumatic brain injury, Neuropsychology, 8, 296-303.
Wagner, D. D., & Heatherington, T. F. (2011). Giving in to temptation: The emerging cognitive
neuroscience of self-regulatory failure. In K. D. Vohs & R. F. Baumeister (Eds.), Handbook of self-regulation: Research, theory, and applications (2nd ed., pp. 41-63). New York: Guilford Press.
Wechsler, D. (2009). Wechsler adult intelligence scale, 4th ed. San Antonio, TX: The
Psychological Corporation. Welsh, M. C., & Pennington, B. F. (1988). Assessing frontal lobe functioning in children: Views
from developmental psychology. Developmental Neuropsychology, 4, 199-230. Weyandt, L. L. (2009). Executive Functioning and Attention Deficit Hyperactivity Disorder.
The ADHD Report, 17(6), 1-7. Weyandt, L., Linterman, I., & Rice, J. (1995). Reported prevalence of attentional difficulties in a
general sample of college students. Journal of Psychopathology and Behavioral
Assessment, 17, 293-304. White, B. P., Becker-Blease, K. A., & Grace-Bishop, K. (2006). Stimulant medication use,
misuse, and abuse in an undergraduate and graduate student sample. Journal of American
College Health, 54, 261–268.
132
Willcutt, E. G., Doyle, A. E., Nigg, J. T., Faraone, S. V., & Pennington, B. F. (2005). Validity of the executive function theory of attention-deficit/hyperactivity disorder: A meta-analytic review. Biological Psychiatry, 57, 1336-1346.
Wolf, L. E. (2001). College students with ADHD and other hidden disabilities: Outcomes and
interventions. In J. Wasserstein, L. E. Wood, R. L., & Liossi, C. (2006). The ecological validity of executive function tests in a
severely brain injured sample. Archives of Clinical Neuropsychology, 21, 429-437.
Woodcock, R.W., McGrew, K.S., &Mather, N. (2001a). Woodcock-Johnson III Tests of
Achievement. Itasca, IL: Riverside. Woodcock, R.W., McGrew, K.S., & Mather, N. (2001b). Woodcock-Johnson III Tests of
Cognitive Abilities. Itasca, IL: Riverside. Zentall, S. S. (2005). Theory and evidence-based strategies for children with attentional
problems. Psychology in the Schools, 42(8), 821-836. Doi: 10.1002/pits.20114 Zentall, S. S., & Zentall, T. R. (1983). Optimal simulation: A model of disordered activity and
performance in normal and deviant children. Psychological Bulletin, 94, 446-471.
133
BIOGRAPHICAL SKETCH
Theodora Passinos Coffman completed her Bachelors of Science degree in Psychology in
2000 at Clemson University in Clemson, South Carolina. She pursued her PhD at the Combined
Doctoral Program in Counseling Psychology and School Psychology at Florida State University
in Tallahassee, Florida. Currently, Theodora is completing an APA-accredited pre-doctoral
psychology internship at GeoCare LLC/South Florida State Hospital in Pembroke Pines, Florida.
She will remain at GeoCare LLC/South Florida State Hospital for her post-doctoral fellowship
once her internship is completed.