Purpose

Dimensional versus Dichotomous Scoring: A Meta-analytical Review of Personality Disorder Stability

James B. Hoelzle, B.S., Erin M. Guell, B.A., & Gregory J. Meyer, Ph.D. The University of Toledo

We conducted a meta-analysis to examine dimensional or dichotomous measurement as a potential moderator of personality disorder (PD) stability over time.

Relevant personality disorders were those in the body or appendix of DSM version III, III-R, IV, or IV-TR, or ICD version 9 or 10. Psychopathy was also included because of its close link to antisocial PD.

Sixty-nine articles met inclusion criteria. From these we coded seventy-five independent or dependent samples. An independent sample presented one set of findings per group of subjects whereas a dependent sample presented different types of data for the same group of subjects (e.g., PD assessed at 6 month and 1 year retest intervals). Findings for instruments in the same method family (i.e., semi-structured interview, questionnaire) were averaged.

Table 1. Descriptive Statistics

The average retest interval was positively skewed (skew = 4.1) so it was re-expressed using the natural log transformation, which normalized the distribution (skew = -.01). The average Log retest interval of 4.7 is equal to 109.95 days.

Methods continued

Introduction

By definition, personality disorders are long-standing, stable dispositions that are inflexible and maladaptive. Since 1980 when the 3rd edition of the Diagnostic and Statistical Manual for Mental Disorders (DSM-III) was published and personality disorders began to be systematically classified on a distinct DSM axis, an extensive research literature developed examining the reliability and validity of personality disorders. However, at present, a systematic and quantitative summary indicating the extent to which personality disorders remain stable over time does not exist. Because stability is a defining feature of personality disorders, the current study fills this important gap in the evidence base.

Furthermore, we focus on dichotomous versus dimensional-measurement of personality disorders as a potential moderator of stability. The relative merits of dichotomous and dimensional measurements is an important one in regards to the creation of DSM-V.

Results continued

Table 2. Sample Size Weighted Average Stability Coefficients

For each cell, k ranges from 6 to 38 and N ranges from 836 to 3877.

To predict expected levels of stability over different time intervals, we used weighted least squares regression analyses, with weights determined by sample size. Length of retest interval was the predictor and separate equations were generated for kappa-based averages across all PDs and the r-based average across all PDs. Because only 11 samples contributed ICC results, a prediction equation was not developed for the ICC findings. Below are the predicted stability values over time.  

Table 3. Predicted Stability For Any Specific PD Over Various Time Intervals

Category Mean SD N 108.3 86.6 Age 31.7 7.8 Retest days 450.0 847.0 Log retest 4.7 1.8

Male 49.4%Type of Measurement 1 Week 1 Year 10 YearsPearson/ Spearman’s r .79 .61 .51Kappa .58 .45 .37

Dichotomous DimensionalAverage Across PDs Kappa ICC r or rS

All Disorders .48 .55 .64Cluster A .41 .54 .64Cluster B .52 .61 .70Cluster C .46 .57 .66

Overall, the magnitude of the stability coefficients is surprisingly low. These findings call into question a key assumption in the DSM and ICD regarding the stability of PD characteristics.

Nonetheless, we expected dimensional variables to be more stable over time than dichotomous diagnoses. The kappa and ICC coefficients in Table 2 show this hypothesis was clearly supported. In line with psychometric expectations, stability values are ordered such that kappa values are lower than ICC values, which in turn are lower than r values. This pattern is evident for the average across all PDs and in each of the three Clusters.

Comparing the two dimensional stability coefficients, mean level changes in PD symptoms likely occurred at follow-up assessment causing ICC results to be less than r results.

Table 2 also suggests that PD stability may vary as a function of the disorders under consideration. Disorders within Cluster B are generally more stable than those in Cluster C, which in turn appear somewhat more stable than those in Cluster A.

Table 3 shows stability clearly decreases as the retest interval lengthens. Regardless of predicted retest interval, however, dimensional stability is higher than dichotomous stability.

A growing amount of empirical research favors the utility of system-cluster dimensions over categorical diagnoses. As a result, the future structure of Axis II in the DSM is in question. Currently, the DSM requires a certain number of criteria to be met in order for a personality disorder diagnosis to be assigned. The diagnosis requires one to make a dichotomous or categorical decision on the presence of a personality disorder even though the underlying criteria form a dimensional measure. However, several authors (Clark & Watson, 1999; Widiger, 2000; Widiger & Clark, 2000) have proposed that for DSM-V it would be more appropriate to quantify Axis II characteristics dimensionally. There are strong psychometric rationales for preferring dimensional over dichotomous scores. When dimensional constructs are artificially dichotomized, it creates a loss of variance (Cohen, 1983) and lower reliability (MacCallum, Zhang, Preacher, and Rucker, 2002). Due to these reasons we expected personality disorders to be more stable over time when they are treated dimensionally rather than dichotomously.

Three types of stability coefficients were examined. All studies examining dichotomous diagnoses reported kappa coefficients, while all studies examining dimensional scores reported either the intraclass correlation (ICC) or Pearson’s/Spearman’s r.

Kappa and the ICC correct for chance agreement and are asymptotically equivalent (i.e., differences are increasingly trivial as N increases). Because of their statistical equivalence, when kappa and ICC results are compared in this meta-analysis it reveals the difference between dichotomous and dimensional PD classifications.

Although both ICC and r can be computed on dimensional scores, r differs from the ICC (and kappa) because it is not lowered by mean differences, only by changes in rank ordering within a distribution. Thus, the ICC (and kappa) provides a more conservative estimate of stability than r.

When considering all three types of coefficients, we expected kappa to be lower than ICC or r because it was computed from an artificially dichotomized dimensional construct. We also expected ICC to be lower than r because it is reduced by differences in both means and rank ordering, while r is reduced only by differences in rank ordering.

Table 2 provides stability values for dimensional and dichotomous PD judgments organized by type of coefficient. Results are presented for any specific PD averaged across all disorders and within Clusters A, B, or C.

Method

To locate relevant studies we conducted a literature search on PsycINFO and PubMed to locate articles with the words "personality disorder" or "Axis II" in conjunction with "stability" or "retest." The initial search yielded 403 articles. To be included in the meta-analysis each article had to meet the following criteria:

1) average age 18-years, 2) sample size 20, 3) retest interval 1 week,4) published in or after 1980, and5) examine 1 specific ICD/DSM disorder or psychopathy.

Clark, L. A., & Watson, D. (1999). Personality, disorder, and personality disorder: Towards a more rational conceptualization. Journal of Personality Disorders, 13, 142-151.

Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of

quantitative variables. Psychological Methods, 7, 19-40.Widiger, T. A. (2000). Personality disorders in the 21st century. Journal of Personality Disorders, 14, 3-16.Widiger, T. A., & Clark, L. A. (2000). Toward DSM-V and the classification of psychopathology. Psychological

Bulletin, 126, 946-963.

Results Discussion

References