On Track Outcomes Program
Client Feedback Form Manual
Prepared for Beacon Health Options by Jeb Brown, PhD, Center for Clinical Informatics
February 2014
For more information about this manual contact Jeb Brown, Ph.D. at
[email protected], or call (801) 541-9720
For more information about the On Track Outcomes Program send email to [email protected]
Beacon Health Options 2 Client Feedback Forms
Table of Contents
Section 1: Overview of the Client Feedback Form ...............................................................................4
Administration .......................................................................................................................................... 4
Outcomes Questionnaires and Clinician Feedback................................................................................... 5
Section 2: Development, Scoring, and Interpretation .........................................................................6
Development............................................................................................................................................. 6
Absenteeism/Presenteeism ...................................................................................................................... 7
Alliance Scale ............................................................................................................................................ 7
Global Distress Scoring and Interpretation............................................................................................... 8
Client Feedback Form Score Ranges for the Three Severity Levels ...................................................... 9
Monitoring Change over Time.................................................................................................................. 9
Case Mix Adjustment and Outcomes Benchmarking ............................................................................. 10
Reliability and Validity............................................................................................................................. 11
Section 4: CFF-Adult Specific Information.........................................................................................13
Reliability and Construct Validity ............................................................................................................ 13
CFF-ADULT Factor Analysis (N=86,185) .............................................................................................. 14
IRT Analysis of Global Distress and Substance Abuse Scales .............................................................. 15
Formulas to predict final score (benchmark score) for the episode of care ...................................... 16
Section 5: CFF – Adolescent Specific Information..............................................................................17
Reliability and Construct Validity ............................................................................................................ 17
Youth CFF Factor Analysis (Responder=Youth, N=9,647) ................................................................... 18
Youth CFF Factor Analysis (Responder= Parent or other adult, N=3747)........................................... 19
Formulas to predict final score (benchmark score) for the episode of care ...................................... 21
Section 6: CFF – Child Specific Information .......................................................................................22
Reliability and Construct Validity ............................................................................................................ 22
Child CFF Factor Analysis (Responder=Youth, N=3,278)..................................................................... 23
Child CFF Factor Analysis (Responder=Parent or other adult, N=6,114) ............................................ 24
Formulas to predict final score (benchmark score) for the episode of care ...................................... 25
References............................................................................................................................. ..........26
Appendix A: Adult CFF Normative and Benchmarking Data ..............................................................28
Results of GLM prediction - adding variables incrementally ............................................................. 29
Beacon Health Options 3 Client Feedback Forms
Appendix B: Youth Self-Report CFF Normative and Benchmarking Data ............................................30
Results of GLM prediction - adding variables incrementally ............................................................. 31
Appendix C: Youth - Parent CFF Normative and Benchmarking Data .................................................32
Results of GLM prediction - adding variables incrementally .............................................................. 33
Appendix D: Child - Self CFF Normative and Benchmarking Data .......................................................34
Results of GLM prediction - adding variables incrementally .............................................................. 35
Appendix E: Child - Parent CFF Normative and Benchmarking Data ...................................................36
Results of GLM prediction - adding variables incrementally ............................................................. 37
Beacon Health Options 4 Client Feedback Forms
Section 1: Overview of the Client Feedback Form
The Client Feedback Forms (CFFs) used in Beacon’s On Track Outcomes Program are brief, reliable and
valid client completed questionnaires designed to measure improvement in symptoms, quality of social
relations, and functioning at work/school and other daily activities. Use of these or similar outcome
questionnaires has been shown to significantly improve treatment outcomes across large sample of
clinicians treating thousands of clients.
The questionnaires measure symptom severity and improvement across a broad range of problems, and
are not intended to be diagnostic or to substitute for clinical evaluation or other assessments that a
clinician may routinely conduct.
There are three versions of the CFF now available:
Adult version (age 18 and older)
Youth version (ages 13-17; completed by youth or adult)
Child version (under age 13; completed by youth or adult)
All versions of the questionnaires have high reliability (coefficient alpha => .87). Extensive factor
analyses demonstrate high construct validity, with items loading on the common factor found in the
most commonly used measures of treatment outcomes.
The Client Feedback Forms are unique among available outcome measures in that items asking for
feedback on the working alliance are included on every questionnaire. The use of alliance items has
been shown to reduce no shows and premature termination while contributing significantly to
improved outcomes.
Administration
The questionnaires were designed for routine use in clinical practice and can be completed by
most clients in less than two minutes, using a simple paper form.
When first introducing the questionnaire to a client, it is best practice to provide a brief
explanation of the reason for the questionnaire, and how it will be used as a routine part of
treatment. At many sites, this explanation is provided by office staff when the client checks in.
Following are a sample explanation scripts used by support staff:
Beacon Health Options 5 Client Feedback Forms
“Please take a moment to fill out the questionnaire. This will help you and your therapist
talk about how treatment is going.”
“We are really excited about the questionnaires. Research shows that therapists who use
questionnaires like these get much better outcomes.”
“Please answer as honestly as you can. This is important to your treatment, because it will
help your therapist understand how to help you.”
Research asking clients to provide feedback on their experience with the questionnaires
indicated a high level of honesty and willingness to complete the alliance items, especially if
they perceived that the clinician was interested in their responses. Of this group, 95% agreed
that the questionnaires were helpful in treatment.
For this reason it is recommended that the clinician acknowledges and value the client’s
willingness to give feedback. Taking a few moments at the start of the session to review the
questionnaire provides the clinician with a wealth of clinically relevant information while
acknowledging the value of the client’s time to complete the questionnaire.
Research of the past several years has revealed that clinicians who use the questionnaires with
a high percentage of their practice display significant gains in overall outcomes.
Outcomes Questionnaires and Clinician Feedback
A large body of research supports the proposition that routine use of outcomes questionnaires
combined with feedback to the clinician results in significant improvement in treatment
outcomes, as measured by pre-post change, percentage of patients improved, and reduced
dropout rates (Lambert, 2009; Goodman et al., 2013).
Among users of On Track Client Feedback Forms and related ACORN questionnaires, observation of frequency of measurement and use of the Decision Support Toolkit provides real world confirmation of results from clinical trials. Both the number of clients measured and the frequency at which the clinician views the data are strongly correlated with year-to-year improvement in treatment outcomes at the clinician level (r> .3; p<.0001; see Brown 2013 in references).
Beacon Health Options 6 Client Feedback Forms
Section 2: Development, Scoring, and Interpretation Development
The CFFs were developed for Beacon’s On Track Outcomes program as part of a suite of forms
developed cooperatively through A Cooperative Online Resource Network (ACORN). The
network consists of statisticians and researchers affiliated with the Center for Clinical
Informatics along with over 1,000 clinicians using client completed questionnaires in a wide
variety of clinical settings across the country.
The initial work on developing the ACORN questionnaire items was done in collaboration with
Warren Lambert, Ph.D., at Vanderbilt University. Dr. Lambert was instrumental in the
development of the Peabody Treatment Progress Battery (PTPB) for adolescents. Items from
the PTPB were utilized with permission, and additional items for adults, adolescents and
children were added using the same item format. Normative data on these items was collected
through the ACORN network of clinicians. Takuya Minami, Ph.D., of the University of
Massachusetts Boston, further assisted in the analyses of the psychometric properties of all of
the items.
The ACORN form development process takes advantage of the network’s ability to collect large
amounts of data and continuously test and refine items over time. Rather than focus on the
development of static forms, this process centers on the development of reliable and valid
items that can be combined flexibly. The psychometric properties of each item, as well as the
unique combination, are carefully evaluated with items for a specific questionnaire selected
based on the population to be measured and the needs of the various participating
organization. The result is a large item inventory with data from over 200,000 clients.
Evidence of the validity and reliability of the CFF was derived from data on tens of thousands of
administrations in both clinical and community settings. The development process included
item analyses to determine:
Item frequencies and distributions
Item correlations
Factor structure
Construct validity
Scale reliability
Sensitivity to change The CFFs consists of items well-suited for general use in outpatient settings. The questionnaires
are designed to be as brief as possible while retaining excellent psychometric properties. The
Beacon Health Options 7 Client Feedback Forms
Adolescent and Child versions may be completed either by the youth or by an adult who knows
the youth well.
Absenteeism/Presenteeism
The two absenteeism/presenteeism items on the adult form are based on the format and
wording of items in the Health and Productivity Questionnaire (HPQ), a questionnaire in the
public domain developed by Kessler and colleagues at Harvard University. Because the CFF is
designed for use by mental health professionals in a mental health setting, the HPQ wording
has been revised to focus on absenteeism/presenteeism specifically due to mental health
problems, rather than both mental and physical health problems encompassed by the original
HPQ questions.
Alliance Scale
All ACORN questionnaires also include items asking the client to provide feedback on their
experience of the prior session. The use of these so-called Alliance items ask for feedback on
elements of the therapeutic working alliance, such as agreement on treatment goals, and the
client’s perception of the quality of the relationship. Use of alliance measure has been
demonstrated to reduce treatment dropout and improve outcomes. Among the large sample
of clinicians using the CFFs, about 80% of clients complete these items routinely. These clients
have significantly better outcomes than those who fail to complete the items.
The client’s responses on these items may be easily influenced by their perception of the
clinician’s response. Clients may be reluctant to give anything other than perfect ratings in
order to avoid hurting the clinician’s feelings or out of fear of possible consequence to the
clinician if “Satisfaction” ratings are high.
It rests on the skill of the clinician to create a therapeutic environment that encourages honest
feedback on the alliance items. Skilled clinicians are about to use the alliance items as tools to
foster a strong collaborative working alliance with the client.
Consistent ratings of near perfect alliance are NOT associated with the best outcomes. Rather,
clients who provide meaningful feedback early in therapy are very likely to rate the alliance as
improved over time. This pattern, displayed by approximately one third of clients, is associated
with significantly greater improvement in treatment.
The best outcomes are associated with improvement on the Alliance Scale over the course of
the treatment episodes. This means that the patients with the best outcomes are also willing
to give feedback that the treatment encounters early in the treatment episode are less than
perfect, otherwise there is no room for improvement.
Beacon Health Options 8 Client Feedback Forms
Failing to complete the alliance items is associated with less improvement. If the client leaves
the items blank, this provides the clinician with an opportunity to initiate a discussion of how
the client is experiencing the treatment process.
Global Distress Scoring and Interpretation
The core global distress scale (GDS) is scored as the mean of all non-missing items on the form.
If more than 4 items are missing from the adult global distress scale or 6 items from the child or
youth scales, the questionnaire is not scored.
Scores can be divided into three severity ranges (Normal, Moderate, Severe) based on
normative data from clinical and community samples. The cut-off scores for each range are
included with the information on the individual questionnaires.
In the case of the adult questionnaire, the cut-off score for the Normal Range was determined
by collecting a sample of over 1,000 individuals from the community who had never sought
mental health service combined with a sample of over 75,000 individuals receiving mental
health services. The cut-off score was calculated using the method proposed by Jacobson &
Truax (1991) as represented by this formula:
C = (SD1)(mean2)+(SD2)(mean1)
SD1+SD2
A score in the normal range means that the score is in a range typical of respondents from a
community sample that have never sought mental health services. Seventy-five percent (75%)
of a community sample and 25% of a clinical sample will fall into this range. Mental health
clients with scores in this range tend to not show improvement with treatment.
A score in the moderate range of distress is characteristic of individual seeking mental health
services. About 50% of a clinical sample will fall into this range. Mental health clients with
scores in this range tend to show significant improvement within a few sessions of therapy, and
most complete treatment with a good outcome in fewer than 8 sessions.
Twenty five percent (25%) of a clinical sample will have scores in the severe range, while fewer
than 10% of a community sample will fall in this range. Clients with scores in this range are
highly likely to show rapid improvement with psychotherapy, but may need more sessions to
realize the full benefit of treatment.
Beacon Health Options 9 Client Feedback Forms
In the case of child and adolescent questionnaires, cut-off scores were estimated by the 25th
and 75th percentile. Practicality prevented collection of large enough community samples.
However, the cut-off scores are comparable to those reported for other similar measures with
the OQ-45 for adults and YOQ-64 for children and youth. Items were selected for the Client
Feedback Form in order to assure comparability to the OQ-45 and YOQ-64. Analysis of archival
data for the OQ-45 and YOQ-64 confirms that the Client Feedback Forms produce results very
similar to these OQ measures.
Client Feedback Form Score Ranges for the Three Severity Levels
Form Normal Range Moderate Range Severe Range Adult CFF 0 to 1.5 1.6 to 2.5 2.6 to 4.0 Adolescent CFF
(Youth completed) 0 to 1.0 1.1 to 1.8 1.9 to 4.0
Adolescent CFF (Parent/Adult completed)
0 to 1.2 1.3 to 1.9 2.0 to 4.0
Child CFF (Youth completed)
0 to 1.0 1.1 to 1.8 1.9 to 4
Child CFF (Parent/Adult completed)
0 to 1.2 1.3 to 1.9 2.0 to 4.0
Monitoring Change over Time
The ability for the clinician to monitor client change as the treatment unfolds is one of the
features of On Track that leads to improved outcomes. Identification of clients who are “off
track” assists the clinician in preventing early drop-out in treatment.
The On Track Decision Support Toolkit provides graphs for each case. The actual client scores
are compared to a “predicted score” at each assessment point. The predicted score is
determined using a statistical prediction technique known as General Linear Modeling. The
method takes advantage of all of the normative data for other clients completing multiple
questionnaires at multiple points in treatment.
The predicted score at each assessment is computed using the initial CFF Global Distress Score,
the assessment number, and the number of weeks that have passed since the initial
assessment to determine the expected assessment at each measurement point. The actual
score can them be compared to the predicted score in order to determine the extent that the
clients current score deviates from the expected score. Clients who scores are significantly
higher than expected at classified as “off track”.
Beacon Health Options 10 Client Feedback Forms
Case Mix Adjustment and Outcomes Benchmarking
Simply measuring pre-post change on an outcome questionnaire provides little information
without some basis for comparison or benchmarking. The On Track program is based on a long
history of research using real world effectiveness data as well as meta-analyses of clinical trials
to establish efficacy.
At the most basic level, change scores on the CFF questionnaires are converted to effect size,
based on dividing the pre-post global distress change score by the standard deviation of the
global distress scores at intake. A simple effect size is calculated by dividing the pre-post change
score by the standard deviation of the outcome measure at intake. An effect size of one means
that the client improved one standard deviation on the measure.
In order to make results comparable to results from clinical trials, effect size is only calculated
for cases with intake scores above the clinical cutoff score. In a general outpatient population,
75% of cases will be in the clinical range. Another reason for excluding non-clinical range cases is that these who enter treatment
reporting little distress do not show improvement on average. This would have the effect of
artificially lowering effect sizes and making comparisons to results from published studies
invalid.
The second manner in which outcomes are benchmarked is by using a large and diverse
normative sample of over 90,000 outpatient treated at hundreds sites around the country. The
sample represents a wide range of ages and ethnic groups, includes those covered by
commercial insurance, EAP programs, Medicaid/Medicare, as well as self-pay and other sources
of funding. As such, the sample is highly representative of patients seen in outpatient general
practice.
In order to establish a benchmark, the statistical procedure known as General Linear Model was
applied to establish which variables collected at intake predicted the final global distress score at
the end of the episode.
Employing this model, it us evident that the first global distress score is the strongest predictor
of subsequent scores, including the final score. The intake score alone accounts for 30% to 50%
of the variance in final scores, depending on the length of treatment.
A second predictor in naturalistic data such as On Track is the session at which the first
questionnaires were administered. If the first assessment is at intake or no later than the
second appoint, measured change is larger than if the first assessment is later. In many cases,
the session number for the first assessment is unknown. In order to take session number into
account, each episode is classified as Early Assessment (session 1 or 2), Later Assessment, or
Beacon Health Options
11 Client Feedback Forms
Unknown. These three categories are included in the General Linear Model as a class variable,
but explain less than 1% of additional variance.
A third potential predictor is diagnosis. Inclusion of diagnosis adds minimally to the prediction
of final score, typically explaining less than .005% of additional variance. In some instances,
such as the youth self-reported version of the CFF, diagnosis is a non-significant predictor.
Diagnosis is not currently included in the On Track benchmarking models.
The prediction of the final (benchmark) score for the On Track program is based on predictive
formulas. These formulas take the form of a simple regression formula: Last Score = First Score
* slope + intercept). The regression formulas are coded from each version of the questionnaire,
with regression formulas differing based on session number at first assessment. Sections 4-6
contain form-specific analyses, including the specific formulas used for each CFF form and
session number.
The appendices provide detailed normative information for each questionnaire with regards to
mean intake scores, final scores, change scores, number of assessments, and average time pre-
post (reported in weeks) for the entire sample, including a breakout by diagnostic group. A
second set of normative information is provided for only those cases with intake scores in the
clinical range.
Each patient’s actual final score is compared to the benchmark target to determine to what
extent the patient reported more or less improvement than comparable patient in the
complete normative sample. The results of the General Linear Model analyses from which the
formulas are derived are provided in the appendices.
The general methodology for benchmarking outcomes has been published in a series of peer
reviewed journal articles, though in these publications the benchmark scores are calculated
using a multivariate GLM. These publications are available upon request (Minami et al. 2007;
Minami et al. 2008a and 2008b; Minami et al. 2011).
Reliability and Validity
Reliability of the global distress scale is measured using Cronbach’s alpha, which is a measure of
internal consistency. This is consistent with classical test theory, which seeks to develop single
factor scales with internal consistency.
Reliability for the CFFs are as follows:
Adult version = 0.89
Youth version = 0.87
Child version = 0.90
Beacon Health Options 12 Client Feedback Forms
Validity of the scales is estimated primarily as construct validity, addressing whether the
questionnaire measures a single construct or factor. Prior research indicates that items on most
outcomes questionnaires used in behavioral health correlate highly with one another and
measure a single construct, generally referred to as “Global Distress.”
For example, Brophy et al. (1988) found that the SCL-90 subscales all load on a common factor, and likewise correlate highly with similar scales from other measures. Miller et al. (2003) found that the Outcome Rating Scale correlates highly with the OQ-45.
Enns et al. (1998) performed factor analyses on the Beck Depression Inventory and the Beck Anxiety Inventory. To quote from this study:
" [T]he parameter estimate was very high (0.784) and a unidimensional, single-factor model of negative affectivity approached the criteria for good fit. It was concluded that the Beck Anxiety and Depression Inventories assess distinct anxiety and depression phenomena to a limited extent when used in a clinically depressed sample."
Recent research by the ACORN collaboration further investigates the relationship between
items assessing states of emotional well-being and high life satisfaction with measures of
psychiatric symptoms and lost productivity (Brown & Minami, 2013). This work demonstrates
that measures of well-being and life satisfaction likewise correlate highly with the same
common factor as symptoms and lost productivity.
The existence of a global distress factor and the fact that multiple outcome questionnaires,
including widely used measures of depression, are all found to be correlated with one another
provide strong evidence of the construct validity of patient self-report outcome measures
designed to measure and assess global subjective distress.
The estimate of reliability and validity for the Alliance Scale is complicated by the fact that the
responses are not normally distributed. Rather, approximately 50% of clients report the alliance
is virtually perfect all of the time, with little variance from week to week. This pattern is the
most common, but is not associated with better than average outcomes. Less than perfect
Alliance Scores at the start of the treatment are associated with better outcomes if the client
remains engaged in treatment. To a large extent the utility of the Alliance Scale is dependent on
the skill of the clinician to elicit and utilize frank feedback from the client. The following sections describe the psychometric properties of each version of the Client
Feedback Form, including detailed information on item analyses.
Beacon Health Options
13 Client Feedback Forms
Section 4: CFF-Adult Specific Information Reliability and Construct Validity
The Adult Client Feedback Form consists of 10 items which assess symptoms of depression,
anxiety, social relationships, and functioning in work and other daily activities. Three additional
items assess problems related to substance abuse. In addition, the forms contain three items
which ask for feedback on the client’s last session experience. Use of these Alliance items is
associated with better outcomes.
The adult form also includes items asking about prior treatment and presence of chronic
illnesses. The variables may be used when performing calculations for case mix adjustment.
The following assessment of the factor structure and reliability of the Adult questionnaire is
based on a sample of 86,185 clients completing the questionnaire at the start of a treatment
episode.
Factor analyses (principal components and varimax rotation) reveal that these items related to
depression, anxiety, interpersonal problems, and impaired functioning in work, school and
other daily activities all load on a common factor labeled Global Distress. Table A displays the
results of this analysis.
Beacon Health Options 14 Client Feedback Forms
CFF-ADULT Factor Analysis (N=86,185)
Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of .54 or greater.
Varimax rotation does little to alter factor structure. Global Distress Scale contains a single factor.
Factor Method: Principle Components
Factor Method: Varimax Rotation
Scale/Subscale Item wording Factor 1 Factor 2 Factor 1 Factor 2
GDS/Symptoms ...feel unhappy or sad? .797 -.096 .801 .055
GDS /Functioning …feel unproductive at work or other daily activities?
.752
-.136
.764
.008
GDS /Symptoms …have problems with sleep (too much or too little)?
.687
-.091
.692
.040
GDS /Symptoms …feel tense or nervous? .735 -.102 .741 .038
GDS /Symptoms …have little or no energy? .744 -.160 .761 -.018
GDS /Symptoms …feel hopeless about the future? .799 -.058 .796 .093
GDS /Social
…have a hard time getting along with family, friends, or coworkers?
.591
.002
.581
.113
GDS /Social …feel lonely .717 -.049 .713 .086
GDS /Symptoms think about harming yourself .543 -.094 .516 .194
GDS /Symptoms …have a hard time paying attention?
.720
-.104
.726
.033
Substance Abuse
…have someone express concerns about your alcohol or drug use?
.257
.839
.096
.873
Substance Abuse
…have had a problem at work, school or home because of alcohol or drug use?
.223
.805
.068
.833
Substance Abuse ...consume five or more drinks on a single occasion
.190
.761
.044
.783
Variance Explained by Each Factor
5.233
2.033
5.121
2.142
Final Commonality Estimate
7.262568
Ten of the items load heavily on the Global Distress Factor. This results in high reliability for the
Global Distress Scale. Reliability as estimated using Cronbach’s coefficient alpha was 0.89.
The CFF-Adult can be scored with subscales for symptoms, interpersonal problems and
functioning. However, these subscales cannot be identified using factor analysis.
Beacon Health Options
15 Client Feedback Forms
The Substance Abuse items do form a separate factor, and are scored on a separate scale with a
Cronbach’s alpha of 0.78. Correlation between the Global Distress Scale and Substance Abuse
scale was .14 (p< .001). The question “how often did you have five or more drinks of alcohol at
one time?” is based on items widely used in assessment of binge drinking behavior, such as
those used in the National Household Survey on Drug Abuse (SAMHSA, 1998) and Harvard
School of Public Health College Alcohol Study (Henry , 1997). The addition of a question that
touches on binge drinking behavior is a valuable supplement.
The evidence of a common factor for global distress demonstrates construct validity. Similar
measures of common symptoms of depression and anxiety have likewise been shown to share
a common factor. These include the Beck Depression Inventory, the Beck Anxiety Inventory,
the OQ-45, and the PHQ9. Correlational studies between the Adult questionnaire and the both
the PHQ9 and the Beck Depression Inventory show a concurrent validity of 0.8.
IRT Analysis of Global Distress and Substance Abuse Scales
The initial development of the Adult form was informed by an IRT analysis of the 10 global
distress scale and 3 substance abuse scale items. These analyses were conducted on a smaller
dataset available at the time of initial form development.
Label
N
Mean
Vari- ance
Kurt- osis
Min
Max
Item- total
MSA
Std. Loading on One Factor
Rasch
Meas- ure
Score
Infit
Out- fit
Disc.
MH (Global Distress)
Unhappy or sad 494 2.4 0.9 -0.6 0 4 0.7 0.91 0.8 39.5 0.6 0.6 1.4
Little or no energy 496 2.1 1.0 -0.5 0 4 0.5 0.94 0.6 44.3 1.0 1.0 0.9
Getting along with family and friends
496
1.6
1.1
-0.3
0
4
0.5
0.94
0.5
52.4
1.2
1.2
0.7
Hard time paying attention
495
1.9
1.2
-0.6
0
4
0.6
0.92
0.7
48.1
0.9
0.8
1.2
Unproductive at work 488 1.7 1.3 -0.7 0 4 0.6 0.91 0.7 50.7 1.0 1.0 1.0
Problems with sleep 490 2.2 1.5 -0.9 0 4 0.6 0.94 0.7 43.1 1.1 1.1 0.9
Tense or nervous 490 2.1 1.4 -0.7 0 4 0.7 0.92 0.7 44.8 0.9 0.9 1.2
Lonely 488 2.0 1.4 -0.8 0 4 0.6 0.92 0.6 46.5 1.2 1.2 0.8
Hopeless 497 1.7 1.3 -0.6 0 4 0.7 0.91 0.8 51.8 0.8 0.8 1.3
Self-harm
490
0.3
0.5
4.7
0
4
0.3
0.90
0.3
78.9
1.7
1.4
0.7
Substance Abuse
Someone expressed SA concerns
488
0.3
0.6
7.5
0
4
0.8
0.7
0.9
46.3
0.9
0.9
1.2
SA problem at work/home
488
0.2
0.4
20.8
0
4
0.6
0.7
0.7
65.8
1.3
1.0
0.9
Five or more drinks 493 0.4 0.7 4.7 0 4 0.7 0.7 0.8 37.9 1.0 1.0 1.0
Beacon Health Options 16 Client Feedback Forms
The table below summarizes the overall scale characteristics based on this sample:
Scale
Items
Cronbach's alpha
MSA
Bentler CFI 1 factor
RMSEA 1 factor
Rasch Person
Reliability
Rasch Item Separation Reliability
MH (Global Distress) 10 0.87 0.92 0.96 0.06 .85-.88 1.00
SA 3 0.84 0.71 1.00 0.00 .16 - .19 0.98
Formulas to predict final score (benchmark score) for the episode of care
First assessment at session 1 or 2 Last GDS score=First GDS Score * 0.5756915262+0.4366357297 (R-square=.30)
First assessment at session 3 or later
Last GDS score=First GDS Score * 0.6236339929 + 0.4950503979 (R-square=.37)
First assessment session unknown Last GDS score=First GDS Score * 0.6088506340 + 0.4065135658 (R-square=.40)
See Appendix A for detailed normative information and predictive models.
Beacon Health Options
17 Client Feedback Forms
Section 5: CFF – Adolescent Specific Information
Reliability and Construct Validity
The Adolescent version of the CFF can be completed by either the youth or a parent. The
questionnaire contains a total of 15 items, seven of which assess symptoms of anxiety and
depression, while eight assess symptoms related to attention and behavioral problems. A
single item inquires about drug or alcohol use.
Factor analysis (principal components) likewise reveals that all items load on a common factor,
as evidenced by reliability (coefficient alpha) of 0.87 for the full scale. The sample size for this
analysis was 13,394. The factor structure was the same whether the questionnaire was
completed by the youth or an adult who knew the youth well.
Factor analysis with varimax rotation indicates the existence of two factors, one containing
symptoms of anxiety and depression (sometimes referred to as internalizing symptoms) while
the second contains symptoms related to attention and behavioral problems (externalizing
symptoms). The main difference is that the internalizing items accounted for slightly more
variance than externalizing items with the youth report version, while the opposite was the
case for the parent completed version.
Beacon Health Options 18 Client Feedback Forms
Youth CFF Factor Analysis (Responder=Youth, N=9,647)
Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of .45 or greater.
Rotated Factor 1 reflects symptoms of depression and anxiety. This accounts for more of the variance with Youth completed forms than Factor 2, which reflects attention and behavioral problems. Factor 3 is substance abuse.
Factor Method:
Principle Components
Factor Method:
Varimax Rotation
Scale
Item wording Factor
1
Factor 2
Factor 3
Factor 1
Factor 2
Factor 3
GDS/Symptoms …eat a lot more or a lot less than usual?
.530
-.115
.081
.461
.224
.196
GDS/Social
…get into trouble?
.520
.527
.088
.011
.649
.368
GDS/Social …interrupt others? .555 .479 -.265 .081 .775 .040
GDS/Social …lie to get things you wanted? .576 .408 .112 .133 .594 .375
GDS/Symptoms …have a hard time controlling your temper?
.644
.180
.040
.342
.510
.268
GDS/Symptoms …worry about a lot of things? .649 -.433 .057 .767 .092 .126
GDS/Symptoms
…feel worthless?
.702
-.403
.094
.783
.131
.183
GDS/Symptoms …have a hard time having fun? .607 -.449 -.057 .751 .099 .004
GDS/Social
…have a hard time waiting your turn?
.580
.315
-.278
.212
.684
-.008
GDS/Social …hang out with kids who get into trouble?
.447
.363
.481
.059
.340
.666
GDS/Social …feel nervous and/or shy around other people?
.530
-.365
-.152
.640
.144
-.083
GDS/Functioning
…lose things you need?
.573
.172
-.162
.302
.538
.060
GDS/Symptoms …have a hard time sleeping because you were worried?
.623
-.425
.053
.743
.082
.117
GDS/Social
…annoy other people on purpose?
.498
.429
-.333
.076
.731
-.053
GDS/Social …think that you don't have any friends?
.594
-.375
-.111
.692
.161
-.030
Substance Abuse …drink alcohol or use other substances
0.284
0.108
0.768
.108
-.045
.818
Variance Explained by Each Factor
5.105
2.181
1.188
3.715
3.182
1.577
Final Commonality Estimate
8.474497
Beacon Health Options
19 Client Feedback Forms
Youth CFF Factor Analysis (Responder= Parent or other adult, N=3747)
Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of .45 or greater.
(Note that Factors 1 and 2 in varimax rotation are reversed from Youth completed version)
Factor 1 reflects attention and behavioral problems and accounts for more variance than Factor 2, symptoms of depression and anxiety.
Factor Method: Principle Components
Factor Method:
Varimax Rotation
Scale
Item wording Factor
1
Factor 2
Factor 3
Factor 1
Factor 2
Factor 3
GDS/Symptoms …eat a lot more or a lot less than usual?
.478
.215
.074
.214
.458
.155
GDS/Social
…get into trouble?
.652
-.483
.031
.782
.019
.220
GDS/Social …interrupt others? .671 -.464 -.212 .841 .059 -.013
GDS/Social …lie to get things you wanted? .647 -.374 .162 .681 .094 .335
GDS/Symptoms …have a hard time controlling your temper?
.707
-.209
-.068
.679
.274
.108
GDS/Symptoms …worry about a lot of things? .531 .551 -.013 .070 .761 .048
GDS/Symptoms
…feel worthless?
.632
.431
.084
.197
.722
.176
GDS/Symptoms …have a hard time having fun? .529 .482 -.000 .108 .704 .067
GDS/Social …have a hard time waiting your turn?
.668
-.345
-.222
.768
.152
-.036
GDS/Social …hang out with kids who get into trouble?
.456
-.198
.602
.326
.090
.704
GDS/Social …feel nervous and/or shy around other people?
.416
.471
-.139
.060
.634
-.092
GDS/Functioning
…lose things you need?
.565
-.028
-.164
.484
.335
-.033
GDS/Symptoms …have a hard time sleeping because you were worried?
.471
.564
.042
.004
.731
.088
GDS/Social …annoy other people on purpose?
.649
-.383
-.152
.761
.107
.032
GDS/Social …think that you don't have any friends?
.536
.409
-.135
.188
.689
-.055
Substance Abuse …drink alcohol or use other substances
.194
-.005
.821
-.040
.068
.840
Variance Explained by Each Factor
5.098
2.414
1.266
3.895
3.423
1.458
Final Commonality Estimate
8.776643
Beacon Health Options 20 Client Feedback Forms
The coefficient alpha for the internalizing factor was 0.84, and for the attention/behavior
problems 0.83. The correlation between the two factors was 0.41. The correlation between the
single substance abuse item and internalizing items was 0.15, while the correlation with
externalizing items was 0.18. All correlations were statistically significant (p< .001).
As with the Adult questionnaire, the Adolescent version is scored with a Global Distress Scale
with subscales for symptoms (anxiety/depression), interpersonal problems, and
attention/functioning items, with the externalizing symptom items divided between
interpersonal and attention/functioning problems.
Despite the evidence for up to three factors as revealed by varimax rotation, the correlation
between factors is strong such that they can be treated as a single factor measure. This is
consistent with scoring instructions for the OQ-64.
As with the Adult questionnaire, the factor analysis shows evidence of construct validity. The
Youth measure will likewise show a strong correlation with other similar youth measures such
as the Youth Outcome Questionnaire, the Child Behavior Checklist, and the Connors Rating
Scale.
Beacon Health Options
21 Client Feedback Forms
Formulas to predict final score (benchmark score) for the episode of care
Youth self-completed First assessment at session 1 or 2
Last GDS score=First GDS Score * 0.5756915262+0.4366357297 (R-square=.30) First assessment at session 3 or later
Last GDS score=First GDS Score * 0.6236339929 + 0.4950503979 (R-square=.37) First assessment session unknown
Last GDS score=First GDS Score * 0.6088506340 + 0.4065135658 (R-square=.40) Youth parent-completed First assessment at session 1 or 2
Last GDS Score = First GDS Score * 0.5397387039 + 0.2866221758 (R-square=.28) First assessment at session 3 or later
Last GDS Score = First GDS Score * 0.6358378356 + 0.2928912887 (R-square=.41) First assessment session unknown
Last GDS Score = First GDS Score * 0. 6439081553 + 0.3105141663 (R-square=.41)
See Appendices B and C for detailed normative information and predictive models.
Beacon Health Options 22 Client Feedback Forms
Section 6: CFF – Child Specific Information
Reliability and Construct Validity
The Child version has 16 items, with similar content to the adolescent version. Some items are
substituted to be more age appropriate. The sample size of the Child version was 8,948. As
with the adolescent version, the factor structure was the same whether completed by the child
or a parent.
Factor analysis (principal components) likewise reveals that all items load on a common factor,
as evidenced by reliability (coefficient alpha) of 0.90 for the full scale.
Factor analysis with varimax rotation indicates the existence of three factors, one containing
symptoms of anxiety and depression (sometimes referred to as internalizing symptoms), the
second with items reflecting behavior problems, and the third with symptoms related to
attention problems. The coefficient alpha for the anxiety/depression factor was 0.75, for the
behavior problems 0.89, and for the attention problems 0.81. The correlation between
behavior problems and attention problems was 0.69, while the correlation between behavior
problems and symptoms of depression/anxiety was 0.42. The correlation between attention
and depression/anxiety was 0.40. All correlations were statistically significant (p<.001).
The questionnaire can be scored as a single Global Distress Scale. Despite the evidence for up
to three factors as revealed by varimax rotation, the correlation between factors is sufficiently
strong such that the instrument can be treated as a single factor measure.
As with the Youth questionnaire, the factor analysis is evidence of construct validity. The Child
measure will likewise show a strong correlation with other similar youth measures such as the
Youth Outcome Questionnaire, the Child Behavior Checklist, and the Connors Rating Scale.
Beacon Health Options
23 Client Feedback Forms
Child CFF Factor Analysis (Responder=Youth, N=3,278)
Factor 1 in principle components is the Global Distress Scale. Note that all but one GDS items have positive factor loadings of .49 or greater. A single item, shyness, is at .25
Rotated Factor 1 reflects behavioral problems. Rotated Factor 2 reflects internalizing
symptoms of depression and anxiety. Rotated Factor 3 reflects problems with attention.
Combined Rotated Factors 1 and 3 for measure of externalizing symptoms.
Factor Method:
Principle Components
Factor Method:
Varimax Rotation
Scale
Item wording Factor
1
Factor 2
Factor 3
Factor 1
Factor 2
Factor 3
GDS/Symptoms …cry easily? 0.499 0.479 -0.120 0.226 0.659 0.079
GDS/Symptoms …feel unhappy or sad? 0.522 0.581 -0.150 0.218 0.763 0.049
GDS/Social …get into trouble? 0.716 -0.316 -0.214 0.764 0.057 0.267
GDS/Social …interrupt others? 0.700 -0.297 0.133 0.550 0.036 0.541
GDS/Social
…lie to get things you wanted?
0.670
-0.242
-0.081
0.626
0.090
0.338
GDS/Symptoms …have a hard time controlling your temper?
0.742
-0.058
-0.229
0.684
0.210
0.221
GDS/Symptoms …worry about a lot of things? 0.485 0.605 0.033 0.079 0.752 0.174
GDS/Symptoms
…have a hard time sitting still?
0.627
-0.103
0.482
0.221
0.145
0.752
GDS/Symptoms …have a hard time paying attention?
0.672
-0.081
0.444
0.266
0.189
0.741
GDS/Social …have a hard time waiting your turn?
0.699
-0.223
0.223
0.468
0.093
0.601
GDS/Social … get into fights with family members and/or friends?
0.674
-0.045
-0.388
0.720
0.295
0.050
GDS/Social …feel nervous and/or shy around other people?
0.3450
0.576
0.093
-0.044
0.659
0.154
GDS/Functioning …lose things you need? 0.555 0.046 0.456 0.122 0.248 0.664
GDS/Social …argue with adults? 0.707 -0.213 -0.330 0.192 0.608 0.158
GDS/Social …annoy other people on purpose?
0.668
-0.266
-0.187
0.694
0.077
0.254
GDS/Social …think that you don't have any friends?
0.493
0.432
-0.037
0.192
0.608
0.158
Variance Explained by Each Factor
6.162
1.888
1.152
3.850
2.732
2.620
Final Commonality Estimate
9.202401
Beacon Health Options 24 Client Feedback Forms
Child CFF Factor Analysis (Responder=Parent or other adult, N=6,114)
Factor 1 in principle components is the Global Distress Scale. Note that all but one GDS items have positive factor loadings of .49 or greater. A single item, shyness, is at .25
Factor structure is same as for Child completed, but with different order in factors. Rotated Factor 1 reflects behavioral problems. Rotated Factor 2 reflects problems with
attention. Rotated Factor 3 reflects internalizing symptoms of depression and anxiety. Combined Rotated Factors 1 and 2 for measure of externalizing symptoms.
Factor Method:
Principle Components
Factor Method:
Varimax Rotation
Scale
Item wording Factor
1
Factor 2
Factor 3
Factor 1
Factor 2
Factor 3
GDS/Symptoms …cry easily? 0.509 0.432 -0.019 0.283 0.140 0.588
GDS/Symptoms …feel unhappy or sad? 0.521 0.615 --0.114 0.306 0.018 0.754
GDS/Social
…get into trouble?
0.765
-0.270
-0.201
0.758
0.351
0.022
GDS/Social …interrupt others? 0.771 -0.210 0.095 0.563 0.566 0.105
GDS/Social …lie to get things you wanted? 0.680 -0.126 -0.057 0.570 0.372 0.135
GDS/Symptoms …have a hard time controlling your temper?
0.758
-0.051
-0.289
0.754
0.214
0.214
GDS/Symptoms …worry about a lot of things? 0.765 -0.270 -0.201 0.758 0.351 0.028
GDS/Symptoms
…have a hard time sitting still?
0.652
-0.249
0.471
0.248
0.802
0.055
GDS/Symptoms …have a hard time paying attention?
0.669
-0.175
0.521
0.212
0.829
0.134
GDS/Social …have a hard time waiting your turn?
0.766
-0.208
0.160
0.518
0.614
0.110
GDS/Social … get into fights with family members and/or friends?
0.724
-0.008
-0.411
0.796
0.088
0.230
GDS/Social …feel nervous and/or shy around other people?
0.253
0.638
0.210
-0.101
0.114
0.701
GDS/Functioning
…lose things you need?
0.581
-0.049
0.446
0.248
0.802
0.055
GDS/Social
…argue with adults?
0.739
-0.115
-0.350
0.794
0.175
0.143
GDS/Social …annoy other people on purpose?
0.725
-0.143
-0.256
0.732
0.249
0.120
GDS/Social …think that you don't have any friends?
0.488
0.416
0.076
0.211
0.207
0.575
Variance Explained by Each Factor
6.576
1.929
1.278
4.219
2.981
2.583
Final Commonality Estimate
9.782697
Beacon Health Options
25 Client Feedback Forms
Formulas to predict final score (benchmark score) for the episode of care
Child self-completed First assessment at session 1 or 2
Last GDS Score = First GDS Score * 0.5978052359 + 0.2011899101 (R-square=.31) First assessment at session 3 or later
Last GDS Score = First GDS Score * 0.5801250274 + 0.3190068320 (R-square=.33) First assessment session unknown
Last GDS Score = First GDS Score * 0.6421563466 + 0.2613480855 (R-square=.40) Youth parent-completed First assessment at session 1 or 2
Last GDS score = First GDS Score * 0.5558274873 + 0.5556823422 (R-square=.31) First assessment at session 3 or later
Last GDS Score = First GDS Score * 0.5809541812 + 0.6167070919 (R-square=.36) First assessment session unknown
Last GDS Score = First GDS Score * 0.6515411875 + 0.3669654397 (R-square=.40)
See Appendices D and E for detailed normative information and predictive models.
Beacon Health Options 26 Client Feedback Forms
References
Brophy CJ, Norvell NK, Kiluk DJ (1988) An Examination of the factor structure and convergent
and discriminant validity for the SCL-90R in an outpatient clinic population. Journal of
Personality Assessment 52(2) 334-340.
Brown (2013) Measurement Plus Feedback Equals Improved Outcomes: An evidence based
practice.
https://psychoutcomes.org/bin/view/DecisionSupportToolkit/ToolkitUsageAndOutcomes
Brown J, Minami T (2013) Quality of Life and Well-being Questionnaires.
https://psychoutcomes.org/bin/view/Questionnaires/QualityOfLife
Enns MW , Coxa BJ, Parker B JDA, & Guertinc JE (1998) Confirmatory factor analysis of the Beck Anxiety and Depression Inventories in patients with major depression Journal of Affective Disorders 47 195-200.
Goodman JD, McKay JR, DePhilippis D (2013) Progress Monitoring in Mental Health and
Addiction Treatment: A Means of Improving Care. Professional Psychology: Research and
Practice 44(4) 231-246.
Henry W. (1997) Harvard School of Public Health College Alcohol Study, Ann Arbor, MI: Inter-
university Consortium for Political and Social Research.
http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/3163
Jacobson, NS., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12-19.
Lambert MJ (2009) Yes, It Is Time for Clinicians to Routinely Monitor Treatment Outcome. In
Miller, S. & Hubble, (M. Eds.), Heart and Soul of Change (2nd ed.). American Psychological
Association Press: Washington.
Miller DD, Duncan Bl, Brown J et al. (2003) The Outcome Rating Scale: A preliminary study of
the reliability, validity , and feasibility of a brief visual analog measure. Journal of Brief Therapy
2 (2), 91-100.
Beacon Health Options
27 Client Feedback Forms
Minami T, Brown GS, McCulloch J, Bolstrom B. (2011) Benchmarking therapists: Furthering the benchmarking method in its application to clinical practice. Quality & Quantity. 46:1699-1708.
Minami T, Wampold BE, Serlin RC, Hamilton EG, Brown GS, & Kircher JC. (2008a) Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment: A preliminary study. Journal of Consulting and Clinical Psychology. 76, 116-124
Minami, T, Serlin, RC, Wampold, BE, Kircher, JC, & Brown, GS (2008b) Using clinical trials to benchmark effects produced in clinical practice, Quality and Quantity 42:513-525.
Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007) Benchmarks for psychotherapy efficacy in adult major depression, Journal of Consulting and Clinical Psychology, 75, 232-243.
SAMHSA (1998) National Household Survey on Drug Abuse.
http://www.samhsa.gov/data/nhsda/98mf.pdf
Beacon Health Options 28 Client Feedback Forms
Appendix A: Adult CFF Normative and Benchmarking Data
Adult CFF - Cases with pre-post change and with first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Personality disorder
Psychosis
Substance abuse
23,774
400
3,173
2,179
69
828
4,728
57
10,473
516
994
10
106
241
2.14 (0.77) 1.64 (0.80) 0.50 (0.73) 5.37 (6.33) 14.02 (18.25)
2.04 (0.72) 1.59 (0.68) 0.45 (0.70) 4.66 (4.17) 13.29 (14.84)
1.91 (0.70) 1.46 (0.72) 0.44 (0.70) 5.04 (5.21) 13.00 (16.34)
2.15 (0.69) 1.68 (0.75) 0.44 (0.71) 5.68 (6.40) 15.27 (18.96) 1.72 (0.72) 1.34 (0.79) 0.38 (0.70) 4.12 (2.44) 10.33 (8.70)
2.46 (0.76) 1.97 (0.82) 0.49 (0.82) 6.82 (9.64) 20.28 (25.08)
2.44 (0.69) 1.84 (0.79) 0.60 (0.75) 6.14 (7.00) 16.76 (20.44)
2.13 (0.77) 1.64 (0.81) 0.49 (0.69) 6.12 (4.98) 16.58 (17.75)
2.05 (0.79) 1.54 (0.78) 0.51 (0.71) 4.82 (5.84) 11.92 (16.39)
2.16 (0.78) 1.74 (0.82) 0.42 (0.71) 5.54 (6.10) 14.56 (17.29)
2.38 (0.73) 1.94 (0.81) 0.44 (0.72) 6.69 (7.93) 17.74 (21.33)
2.50 (0.61) 1.83 (0.66) 0.67 (0.52) 4.30 (2.50) 8.37 (3.95) 2.24 (0.84) 1.86 (0.99) 0.37 (0.83) 6.51 (6.27) 21.57 (25.21)
1.72 (0.94) 1.35 (0.82) 0.38 (0.83) 5.73 (5.34) 14.41 (15.00)
Adult CFF - Cases with intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Personality disorder
Psychosis
Substance abuse
19,250
320
2,327
1,839
44
751
4,310
46
8,050
423
892
10
88
150
2.41 (0.58) 1.80 (0.75) 0.61 (0.71) 5.56 (6.54) 14.53 (18.79)
2.29 (0.54) 1.72 (0.64) 0.58 (0.66) 4.81 (4.45) 13.93 (15.52)
2.22 (0.52) 1.64 (0.68) 0.58 (0.69) 5.24 (5.45) 13.47 (17.02)
2.35 (0.55) 1.79 (0.73) 0.56 (0.69) 5.73 (6.60) 15.46 (19.38) 2.11 (0.54) 1.58 (0.73) 0.52 (0.66) 3.82 (2.00) 10.80 (8.73)
2.61 (0.62) 2.05 (0.80) 0.56 (0.79) 6.72 (8.38) 20.22 (24.59)
2.56 (0.58) 1.90 (0.77) 0.66 (0.73) 6.20 (7.05) 16.79 (20.37)
2.41 (0.54) 1.85 (0.71) 0.56 (0.70) 5.06 (4.24) 14.62 (17.45) 2.36 (0.58) 1.73 (0.74) 0.63 (0.70) 5.04 (6.20) 12.47 (17.15)
2.41 (0.61) 1.88 (0.78) 0.53 (0.68) 5.77 (6.32) 14.98 (16.90)
2.54 (0.60) 2.04 (0.76) 0.50 (0.70) 6.70 (8.02) 17.74 (21.45)
2.50 (0.61) 1.83 (0.66) 0.67 (0.52) 4.30 (2.50) 8.37 (3.95) 2.49 (0.68) 2.02 (0.96) 0.46 (0.82) 6.89 (6.63) 21.93 (24.82)
2.32 (0.60) 1.65 (0.70) 0.68 (0.73) 6.03 (5.98) 14.40 (17.13)
Beacon Health Options
29 Client Feedback Forms
Results of GLM prediction - adding variables incrementally
Adult Model: First Global Distress Score (GDS) predicts final GDS
This model includes all members, regardless of the timing of their assessments or change status
R-Square = 0.3399
Variable Parameter estimate SE t-Value Pr > |t|
Intercept 0.4368 0.0085 51.12 <.0001
firstgds 0.5917 0.0038 154.15 <.0001 Model: Last GDS Score = 0.5917*First GDS score + 0.4368 (intercept)
F=23761.7, p<.0001
Adult Model: First Score predicts final score, Broken out by first assessment (Early vs. Late)
Early R-square = 0.2998
Variable Parameter estimate SE t-Value Pr > |t|
Intercept 0.4366 0.01515999 28.80 <.0001 First GDS 0.5757 0.0065241 88.24 <.0001
Last GDS score= 0.5757*First GDS Score + 0.4366
Model F=7786.44, p <.0001
Later
R-square = 0.3739
Variable Parameter estimate SE t-Value Pr > |t|
Intercept 0.4951 0.0247 20.08 <.0001 First GDS 0.6236 0.0120 51.84 <.0001
Last GDS score= 0.6236*First GDS Score + 0.4950
Model F=2687.65, p <.0001
Beacon Health Options 30 Client Feedback Forms
Appendix B: Youth Self-Report CFF Normative and Benchmarking Data
Youth Self-Report CFF: All cases with pre-post change and first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Personality disorder
Psychosis
Substance abuse
5,048
260
426
310
90
33
480
4
3,205
140
64
1
5
30
1.43 (0.70) 1.09 (0.68) 0.34 (0.62) 4.35 (3.92) 13.14 (14.56) 1.56 (0.74) 1.13 (0.71) 0.44 (0.63) 4.24 (3.61) 14.23 (14.25)
1.38 (0.65) 0.98 (0.59) 0.39 (0.60) 4.22 (3.93) 12.23 (13.20)
1.33 (0.59) 0.98 (0.62) 0.36 (0.59) 4.70 (3.62) 15.07 (15.15)
1.52 (0.72) 1.06 (0.71) 0.46 (0.68) 3.72 (2.75) 9.78 (9.78) 1.61 (0.84) 1.17 (0.70) 0.43 (0.75) 4.64 (3.12) 14.91 (11.95)
1.60 (0.69) 1.17 (0.69) 0.43 (0.64) 4.52 (3.87) 13.22 (12.64)
1.10 (0.68) 0.53 (0.49) 0.57 (0.73) 5.50 (0.58) 22.61 (15.59)
1.40 (0.70) 1.10 (0.70) 0.30 (0.62) 4.33 (4.02) 13.03 (15.15) 1.46 (0.74) 1.06 (0.70) 0.40 (0.64) 4.42 (4.03) 13.92 (13.95)
1.42 (0.75) 1.08 (0.61) 0.33 (0.54) 4.92 (3.76) 15.04 (14.18)
2.75 (0.0) 2.13 (0.0) 0.62 (0.0) 5.00 (0.0) 14.00 (0.0)
1.34 (0.68) 1.25 (0.78) 0.09 (0.24) 3.20 (2.68) 5.80 (5.89) 1.00 (0.68) 0.78 (0.55) 0.22 (0.44) 3.10 (2.52) 6.60 (6.86)
Youth Self Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder Not specified
Other
PTSD
Personality disorder
Psychosis
Substance abuse
4,107
229
354
260
79
27
424
2
2,537
119
52
1
4
19
1.65 (0.57) 1.22 (0.66) 0.42 (0.63) 4.45 (3.98) 13.46 (14.91)
1.71 (0.65) 1.22 (0.69) 0.49 (0.63) 4.37 (3.76) 14.49 (14.62) 1.57 (0.52) 1.09 (0.56) 0.48 (0.60) 4.19 (3.25) 12.12 (12.02) 1.51 (0.48) 1.08 (0.61) 0.43 (0.60) 4.76 (3.67) 15.07 (15.06)
1.66 (0.64) 1.14 (0.72) 0.52 (0.68) 3.64 (2.79) 9.35 (9.72)
1.86 (0.70) 1.29 (0.68) 0.57 (0.76) 4.48 (3.17) 13.99 (11.51) 1.76 (0.57) 1.27 (0.65) 0.48 (0.66) 4.57 (3.96) 13.50 (12.89) 1.69 (0.09) 0.66 (0.75) 1.03 (0.84) 5.00 (0 .0) 15.36 (11.62)
1.64 (0.56) 1.26 (0.67) 0.39 (0.62) 4.47 (4.17) 13.50 (15.78)
1.65 (0.64) 1.17 (0.69) 0.48 (0.65) 4.54 (4.27) 14.18 (14.70) 1.64 (0.63) 1.24 (0.56) 0.40 (0.56) 4.79 (3.80) 14.92 (14.38) 2.75 (0.0) 2.13 (0.0) 0.62 (0.0) 5.00 (0.0) 14.00 (0.0)
1.54 (0.59) 1.43 (0.77) 0.10 (0.28) 2.00 (0.0) 3.18 (0.71)
1.42 (0.48) 1.01 (0.53) 0.40 (0.42) 3.10 (3.00) 6.07 (7.29)
Beacon Health Options
31 Client Feedback Forms
Parameter estimate SE t-Value Pr > |t| 0.1940 0.0411 4.72 <.0001 0.5601 0.0245 22.87 <.0001
Parameter estimate SE t-Value Pr > |t| 0.1809 0.0506 3.57 0.0004 0.5997 0.0356 16.83 <.0001
Results of GLM prediction - adding variables incrementally
Youth Self-Report Model 1: First Global Distress Score (GDS) predicts final GDS
This model includes all members, regardless of the timing of their assessments or change status
R-Square = 0.3569
Variable Parameter estimate SE t-Value Pr > |t|
Intercept 0.2489 0.0177 14.09 <.0001 First GDS 0.5887 0.0111 52.92 <.0001
Model: Last GDS Score = 0.5887*First GDS score + 0.2489(intercept)
Model F=2800.10, p <.0001
Youth Self-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)
Early
R-square = 0.3216
Variable
Intercept First GDS
Last GDS score= 0.5601*First GDS Score + 0.1940 Model F=523.25, p <.0001
Later
R-square=0.4011
Variable
Intercept
First GDS
Last GDS score= 0.5997*First GDS Score + 0.1809
Model F=283.25, p<.0001
Beacon Health Options 32 Client Feedback Forms
Appendix C: Youth - Parent CFF Normative and Benchmarking Data
Youth - Parent CFF: All cases with pre-post change and first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Personality disorder
Psychosis
Substance abuse
5,988
314
404
302
110
16
373
3
4,205
169
50
3
2
37
1.66 (0.76) 1.34 (0.76) 0.31 (0.66) 4.02 (3.51) 13.13 (15.01) 1.96 (0.83) 1.55 (0.81) 0.41 (0.76) 4.45 (5.06) 15.72 (18.10)
1.47 (0.69) 1.13 (0.72) 0.34 (0.63) 4.00 (3.04) 12.01 (12.03)
1.48 (0.69) 1.16 (0.68) 0.32 (0.63) 3.97 (2.81) 11.96 (11.75)
1.96 (0.81) 1.39 (0.83) 0.57 (0.77) 3.99 (3.30) 12.83 (15.08) 1.64 (0.87) 1.42 (0.97) 0.22 (0.63) 4.06 (1.81) 13.71 (9.49)
1.66 (0.72) 1.26 (0.71) 0.40 (0.64) 4.43 (3.83) 12.50 (13.04)
1.56 (0.45) 1.48 (0.57) 0.08 (0.13) 4.67 (3.78) 10.05 (7.47)
1.66 (0.76) 1.37 (0.76) 0.29 (0.66) 3.92 (3.35) 13.09 (15.24) 1.75 (0.82) 1.41 (0.82) 0.33 (0.72) 4.84 (5.19) 16.17 (17.79)
1.71 (0.73) 1.40 (0.72) 0.31 (0.71) 4.22 (3.07) 12.82 (18.75)
1.79 (0.38) 1.42 (0.79) 0.38 (0.65) 9.33 (8.08) 18.05 (13.79)
2.34 (0.22) 1.38 (0.18) 0.97 (0.04) 9.50 (9.19) 37.21 (23.74) 1.06 (0.65) 0.87 (0.65) 0.18 (0.63) 3.76 (1.82) 9.73 (7.92)
Youth Parent Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder Not specified
Other
PTSD
Personality disorder
Psychosis
Substance abuse
4,346
245
258
201
88
12
269
2
3,091
121
39
3
2
15
2.00 (0.57) 1.55 (0.73) 0.45 (0.66) 4.16 (3.73) 13.71 (15.78)
2.29 (0.60) 1.74 (0.76) 0.54 (0.74) 4.71 (5.48) 16.90 (19.62) 1.88 (0.49) 1.38 (0.69) 0.50 (0.64) 4.03 (3.04) 12.16 (12.32)
1.87 (0.47) 1.37 (0.65) 0.49 (0.62) 4.08 (3.12) 12.22 (12.59)
2.25 (0.62) 1.59 (0.78) 0.66 (0.80) 4.09 (3.49) 13.62 (16.17)
1.98 (0.73) 1.64 (0.98) 0.33 (0.67) 4.08 (1.73) 15.36 (10.44) 2.00 (0.52) 1.44 (0.69) 0.55 (0.65) 4.68 (4.08) 13.14 (13.84)
1.75 (0.44) 1.69 (0.62) 0.06 (0.18) 5.50 (4.95) 9.57 (10.50)
1.99 (0.57) 1.57 (0.72) 0.42 (0.65) 4.03 (3.53) 13.60 (15.88)
2.14 (0.60) 1.68 (0.76) 0.46 (0.74) 5.21 (5.35) 17.71 (17.65) 1.98 (0.53) 1.54 (0.72) 0.44 (0.74) 4.33 (3.33) 13.44 (20.94)
1.79 (0.38) 1.42 (0.79) 0.38 (0.65) 9.33 (8.08) 18.05 (13.79)
2.34 (0.22) 1.38 (0.18) 0.97 (0.04) 9.50 (9.19) 37.21 (23.74)
1.69 (0.39) 1.17 (0.66) 0.52 (0.64) 3.67 (1.91) 7.66 (7.70)
Beacon Health Options
33 Client Feedback Forms
Results of GLM prediction - adding variables incrementally
Youth Parent-Report Model 1: First Global Distress Score (GDS) predicts final GDS
This model includes all members, regardless of the timing of their assessments or change status
R-Square = 0.3864
Variable Parameter estimate SE t-Value Pr > |t| Intercept 0.3079 0.0186 16.58 <.0001 First GDS 0.6250 0.0102 61.35 <.0001 Model: Last GDS Score = 0.3079*First GDS score + 0.6250 (intercept) Model F=3763.84, p <.0001
Youth Parent-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)
Early R-square = 0.2841
Variable
Intercept
Parameter estimate 0.2866
SE 0.0526
t-Value 5.45
Pr > |t| <.0001
First GDS 0.5397 0.0279 19.33 <.0001 Last GDS score= 0.5397*First GDS Score + 0.2866 Model F=373.48, p<.0001
Later
R-square=0.4148 Variable Intercept
Parameter estimate 0.2929
SE 0.0624
t-Value 4.69
Pr > |t| <.0001
First GDS 0.6358 0.0410 15.50 <.0001 Last GDS score= 0.6358*First GDS Score + 0.2929 Model F=240.27, p <.0001
Beacon Health Options 34 Client Feedback Forms
Appendix D: Child - Self CFF Normative and Benchmarking Data
Child - Self CFF, All cases with pre-post change and first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Substance abuse
2,526
135
184
129
59
3
88
2
1,831
68
24
3
1.53 (0.74) 1.22 (0.76) 0.31 (0.66) 4.07 (3.58) 13.14 (14.15)
1.93 (0.76) 1.54 (0.83) 0.39 (0.70) 4.19 (3.62) 16.46 (20.10)
1.40 (0.69) 1.14 (0.68) 0.26 (0.60) 3.37 (1.82) \10.01 (8.49)
1.44 (0.60) 1.07 (0.67) 0.36 (0.61) 4.35(2.90) 15.06 (14.08) 1.71 (0.75) 1.35 ( 0.77) 0.35 (0.68) 3.42 (1.80) 10.78 (9.91)
1.58 (0.84) 1.98 (1.61) -0.40 (1.06) 2.00 (0.00) 8.05 (5.00)
1.94 (0.72) 1.34 (0.86) 0.59 (0.81) 3.91 (2.53) 12.21 (10.60)
0.94 (0.26) 0.50 (0.53) 0.44 (0.80) 8.00 (5.66) 27.36 (13.23)
1.48 (0.73) 1.20 (0.74) 0.29 (0.65) 4.15 (3.84) 13.19 (14.35)
1.81 (0.78) 1.46 (0.91) 0.35 (0.66) 4.06 (3.23) 14.19 (12.69)
1.76 (0.62) 1.42 (0.71) 0.34 (0.75) 3.50 (2.50) 8.42 (8.54)
1.67 (0.34) 0.83 (0.50) 0.83 (0.73) 6.33 (4.04) 27.33 (31.51)
Child Self Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Substance abuse
2,077
128
150
115
56
3
84
2
1,448
64
24
3
1.75 (0.61) 1.36 (0.73) 0.39 (0.66) 4.19 (3.65) 13.63 (14.68)
2.02 (0.69) 1.61 (0.79) 0.40 (0.72) 4.28 (3.70) 16.94 (20.48)
1.61 (0.57) 1.26 (0.64) 0.35 (0.57) 3.53 (1.92) 10.56 (8.71)
1.55 (0.52) 1.13 (0.62) 0.42 (0.55) 4.42 (3.01) 15.00 (14.14)
1.78 (0.69) 1.40 (0.76) 0.39 (0.68) 3.39 (1.84) 11.10 (10.06)
1.58 (0.84) 1.98 (1.61) -0.40 (1.06) 2.00 (0.00) 8.05 (5.00)
2.01 (0.64) 1.40 (0.84) 0.61 (0.82) 3.96 (2.58) 12.46 (10.74) 0.94 (0.26) 0.50 (0.53) 0.44 (0.80) 8.00 (5.66) 27.36 (13.23)
1.74 (0.59) 1.36 (0.72) 0.38 (0.66) 4.28 (3.95) 13.73 (14.99)
1.89 (0.72) 1.52 (0.90) 0.37 (0.67) 4.06 (3.25) 14.27 (12.90)
1.76 (0.62) 1.42 (0.71) 0.34 (0.75) 3.50 (2.50) 8.42 (8.54) 1.67 (0.34) 0.83 (0.50) 0.83 (0.73) 6.33 (4.04) 27.33(31.51)
Beacon Health Options
35 Client Feedback Forms
Results of GLM prediction - adding variables incrementally
Child Self-Report Model 1: First Global Distress Score (GDS) predicts final GDS
This model includes all members, regardless of the timing of their assessments or change status
R-Square = 0.3716
Variable Parameter estimate SE t-Value Pr > |t| Intercept 0.2641 0.0275 9.59 <.0001 First GDS 0.6260 0.0162 38.63 <.0001 Model: Last GDS Score = 0.6260*First GDS score + 0.2641 (intercept) Model F=1492.28 , p<.0001
Child Self-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)
Early
R-square = 0.3108
Variable
Intercept
Parameter estimate 0.2012
SE 0.0869
t-Value 2.32
Pr > |t| 0.0211
First GDS 0.5978 0.0465 12.85 <.0001
Last GDS score= 0.5978*First GDS Score + 0.2012 Model F=165.05 , p<.0001
Later
R-square=0.3338 Variable Intercept
Parameter estimate 0.3190
SE 0.1098
t-Value 2.90
Pr > |t| 0.0043
First GDS 0.5801 0.0676 8.58 <.0001 Last GDS score= 0.5801*First GDS Score + 0.3190 Model F=73.67,p<.0001
Beacon Health Options 36 Client Feedback Forms
Appendix E: Child - Parent CFF Normative and Benchmarking Data
Child Parent Report CFF: All cases with pre-post change and first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks
Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder
Not specified
Other
PTSD
Personality disorder
Substance abuse
4980
313
457
198
145
4
100
1
3570
147
39
1
5
1.88 (0.77) 1.60 (0.78) 0.28 (0.67) 4.12 (3.62) 13.66 (15.08)
2.36 (0.68) 2.01 (0.67) 0.34 (0.75) 4.45 (3.90) 16.57 (18.48)
1.80 (0.70) 1.59 (0.72) 0.21 (0.68) 4.73 (4.46) 14.51 (14.32)
1.82 (0.68) 1.44 (0.68) 0.39 (0.62) 4.29 (3.66) 13.57 (14.65) 2.31 (0.72) 1.86 (0.73) 0.45 (0.63) 4.51 (3.72) 13.98 (14.08)
2.00 (0.98) 1.91 (1.04) 0.09 (0.41) 3.75 (2.36) 12.14 (7.98)
2.06 (0.73) 1.73 (0.71) 0.33 (0.67) 3.87 (2.78) 12.14 (10.75)
2.06 (0.0) 0.81 (0.00) 1.25 (0.00) 6.00 (0.00) 18.00 (0.00)
1.81 (0.77) 1.54 (0.79) 0.27 (0.66) 3.94 (3.36) 13.12 (14.92)
2.05 (0.75) 1.80 (0.71) 0.25 (0.76) 5.47 (5.51) 18.73 (16.90)
2.11 (0.77) 1.79 (0.82) 0.32 (0.62) 4.92 (3.77) 13.64 (11.89)
1.88 (0.0) 2.25 (0.00) -0.38 (0.00) 8.00 (0.00) 22.00 (0.00) 1.46 (0.78) 1.51 (0.90) -0.05 (1.28) 3.40 (1.67) 13.40 (7.36)
Child-Parent Report CFF: Intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2.
N
First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd)
All
ADHD
Adjustment disorder
Anxiety
Behavior disorder
Bipolar
Depression
Eating disorder Not specified
Other
PTSD
Personality disorder
Substance abuse
4,048
298
361
160
132
3
86
1
2,845
125
33
1
3
2.13 (0.60) 1.76 (0.72) 0.38 (0.65) 4.25 (3.77) 14.13(15.61)
2.43 (0.61) 2.04 (0.67) 0.39 (0.73) 4.52 (3.98) 16.88 (18.79)
2.06 (0.53) 1.74 (0.65) 0.32 (0.64) 4.93 (4.82) 15.04 (14.95)
2.05 (0.53) 1.60 (0.61) 0.45 (0.62) 4.46 (3.89) 13.91 (15.32)
2.46 (0.57) 1.96 (0.67) 0.50 (0.62) 4.58 (3.83) 14.22 (14.43)
2.33 (0.89) 2.27 (0.92) 0.06 (0.50) 2.67 (1.15) 9.43 (7.16) 2.25 (0.60) 1.83 (0.69) 0.42 (0.65) 4.09 (2.92) 12.96 (11.22)
2.06 (0.00) 0.81 (0.00) 1.25 (0.00) 6.00 (0.00) 18.00 (0.00)
2.09 (0.59) 1.72 (0.73) 0.38 (0.64) 4.05 (3.50) 13.57 (15.45)
2.26 (0.59) 1.90 (0.65) 0.36 (0.70) 5.43 (5.23) 18.79 (16.74) 2.35 (0.57) 1.95 (0.78) 0.39 (0.65) 5.18 (4.04) 13.91 (12.55)
1.88 (0.00) 2.25 (0.00) -0.38 (0.00) 8.00 (0.00) 22.00 (0.00)
1.98 (0.44) 1.56 (0.22) 0.42 (0.64) 3.00 (1.00) 17.43 (2.97)
Beacon Health Options
37 Client Feedback Forms
Results of GLM prediction - adding variables incrementally
Child Parent-Report Model 1: First Global Distress Score (GDS) predicts final GDS
This model includes all members, regardless of the timing of their assessments or change status
R-Square = 0.3867
Variable Parameter estimate SE t-Value Pr > |t| Intercept 0.4136 0.0228 18.14 <.0001 First GDS 0.6296 0.0112 56.02 <.0001 Model: Last GDS Score = 0.6296*First GDS score + 0.4136 (intercept) Model F=3138.58 , p<.0001
Child Parent-Report Model 2: First Score predicts final score, Broken out by first assessment (Early vs. Late)
Early R-square = 0.3076
Variable
Intercept
Parameter estimate 0.5557
SE 0.0595
t-Value 9.34
Pr > |t| <.0001
First GDS 0.5558 0.0271 20.50 <.0001
Last GDS score= 0.5558*First GDS Score + 0.5557 Model F=420.30, p <.0001
Later
R-square=0.3568 Variable
Intercept
Parameter estimate 0.6167
SE 0.0888
t-Value 6.94
Pr > |t| <.0001
First GDS 0.5810 0.0455 12.77 <.0001 Last GDS score= 0.5810*First GDS Score + 0.6167 Model F=163.08, p<.0001