Copyright 1987 byThe Cerontological Society of America

The validity of older homeowners' reports of the repair status of their homes wasdetermined by comparing their assessments with housing inspection records. Although the

observed agreement is high, the ability of older homeowners to provide valid assessments ofhome repair status is overestimated because most of this agreement is due to chance. In

addition, older homeowners are least likely to provide valid reports when repairs are needed.Given the policy importance of accurate reporting about potentially unsafe dwelling unit

conditions, these results are disturbing.Key Words: Home repair, Validity study, Housing inspection

Validity of Older Homeowners' HousingEvaluations1

Alexander Chen, PhD2 and Sandra Newman, PhD3

Much of what is known about housing conditionsin the nation comes from data collected from hous-ing unit occupants in special purpose surveys con-ducted by individual researchers (e.g., Lawton, 1980;National Council on Aging, 1975; Rabushka & Jacobs,1980) and government agencies (Comptroller Gen-eral, 1977; U. S. Department of Commerce, variousyears; U. S. Department of Housing and Urban De-velopment, 1979). The implicit assumption is thatsuch self-reports are reasonably accurate representa-tions of actual housing conditions and are, there-fore, useful as indicators in a housing demand modelor as elements in a formula for the distribution offederal housing assistance dollars. Although house-hold surveys are not inexpensive, reliance on themfor gathering information on housing conditions isoften dictated by their lower cost relative to alterna-tive means of data collection, such as through dwell-ing inspections by experts. But if self-reports ofhousing conditions are subject to response errors,the value of this attempt at cost containment is surelydebatable.

Surveys of older individuals may be particularlyprone to response errors because older persons maybe more influenced by interviewers than youngerrespondents (Groves & Magilavy, 1980; Herzog,1979), experience greater problems of recall of re-cent events (Botwinick, 1978; Craik, 1977; Sudman &Bradburn, 1974), be less willing to give what theyperceive to be socially undesirable responses (Klein,1972; Klein & Birren, 1972), and be more subject toresponse set effects (Herzog & Rodgers, 1982). Theseconcerns make it particularly worthwhile to examine

""This paper is based on research funded by the U. S. Department ofHousing and Urban Development, Office of Policy Development and Re-search, Contract H-2988. The authors gratefully acknowledge the helpfulcomments of Greg Duncan, Bruce Jacobs, Graham Kalton, Aaron Pallas,Willard Rodgers, and Raymond Struyk.

Assistant Professor, Department of Housing and Design, Room 1401Marie Mount Hall, University of Maryland, College Park, MD, 20742.

3Associate Professor, Senior Research Scientist, Johns Hopkins Univer-sity, Shriver Hall, C-3, Baltimore, MD, 21218.

the accuracy of assessments of housing conditionsgiven by older respondents.

This issue was addressed in a validation study inwhich reports of housing repair status obtained ininterviews with older homeowners were checkedagainst the highly accurate records of repair needcollected by trained housing inspectors. The datawere obtained as part of a larger evaluation study ofthe effects of the provision of home repair serviceson the condition of housing units owned by olderpersons (see Newman et al., 1981). Personal inter-views were conducted with a probability sample of1,027 Baltimore, Maryland homeowners 55 years orolder; and within a few days of the interview, hous-ing rehabilitation experts independently inspectedand recorded the repair needs in over 57% of thesehomes. Analysis revealed no systematic differencesbetween the homes for which records were obtainedand those for which they were not (Newman et al.,1981). Specifically, for each of five housing systemsor features including heating, electrical, plumbing,interior, and exterior, homeowners were askedwhether repairs were needed in their home, and, ifso, "what needs to be done?" Housing experts in-spected these homes and kept detailed records ofneeded repairs. These detailed records permittedprecise measurement of the validity of each respon-dent's report. Homeowner responses that agree withthe inspection records are considered valid.

The validation of any survey response is extremelydifficult to obtain since the design of the validationstudy may yield biased conclusions (e.g., Duncan &Hill, 1985; Marquis, 1981). Three sources of errors ofvalidation have been identified (Duncan & Hill,1985): use of an erroneous validation source; use of adifferent definition in the validation source than inthe survey question; and the presence of coding ortranscription errors between the validation source orinterview information and the data file. Several stepswere taken to address these concerns. First, thehousing inspections were conducted by highlytrained experts with many years of experience in the

Vol. 27, No. 3,1987 309

at University of W

indsor on October 29, 2014

http://gerontologist.oxfordjournals.org/D

ownloaded from

field; their housing inspections resulted in detailedinspection records for each housing unit. Thesecharacteristics maximized the objectivity of the in-spection records and enabled us to regard them astrue measures of repair need. To minimize the po-tential over- or underestimation of response errorsdue to the design of the study, the analysis wasrestricted to only those items with exact matchesbetween the validation source and the respondentreport (36 items). Procedurally, verbatim accounts ofthe repair needs voiced by the older homeownerwere recorded and matched directly with the specifichousing record obtained from the housing inspec-tion (e.g., the homeowner response that the bath-room faucets were leaking was matched with itsidentical counterpart on the inspection record). Thisrestriction eliminated 162 housing items that werementioned by at least one homeowner but were notrecorded in the inspection and 212 items that werenoted on at least one inspection but not mentionedby any homeowner. Because the second total (212) isgreater than the first, it is expected that the bias thatmay result from this truncation of variables is a con-servative one. Finally, the coding of survey re-sponses and expert records underwent extensivescrutiny and were subject to independent checksand verifications.

Past Research

Although studies of various sources of responseerror for sample surveys on a large number of topicsabound in the literature (e.g. Cannell et. al., 1965;Ferber, 1955; Kain & Quigley, 1972; Kish & Lansing,1954; Neter, 1970; Palmer, 1943), including severalthat are focused specifically on the elderly popula-tion (Gergen & Back, 1966; Herzog & Rodgers, 1982),only one study has been published that reports onthe accuracy of older persons' reports on housingconditions. Rabushka and Jacobs (1980) used datafrom a 1975 survey of maintenance and repair activityof elderly homeowners (defined as 60 years of age orolder). Each homeowner was asked to evaluate theoverall adequacy of seven major housing systemsincluding (1) plumbing, (2) electrical, (3) heating, (4)roofing, (5) structure and foundation, (6) house exte-rior, and (7) kitchen appliances. In addition, theywere asked to assess the repair needs of specifichousing features. The interviews were followed byindependent evaluations of repair need by trainedobservers. A comparison of the aggregated responsesprovided the basis for the analysis. In general, thesample of elderly homeowners reported adequateconditions despite independent observations thatrepairs were needed for the major housing systemsand 28 separate housing items under review.

The full significance of this finding is difficult toassess, however, for three reasons. First, it is notobvious how to interpret comparisons between ade-quacy and repair need: The two concepts can con-ceivably measure two different attributes. Second,the authors compared only the aggregate propor-tions of observers and homeowners who reported

repair needs. These aggregates may hide consider-able variation not only between the specific housingitems but also between types of agreement, namely,agreement that repairs are needed or are notneeded. As noted later, the incidence of elderlyowners disagreeing with experts that a repair isneeded is of particular policy concern. Third, acomparison of two aggregate proportions did notindicate whether the results would be expected bychance; that is, if the relationship between re-sponses was statistically independent so that the cellprobabilities equal the product of the marginal prob-abilities (Davis, 1971).

Analysis

In this analysis, two statistical measures of agree-ment were relied upon, Cohen's Kappa and theColeman-Light measure of conditional agreement,to evaluate the validity of homeowner reports re-garding repair status. (See Liebetrau, 1983 for a con-cise discussion of these two measures.) These mea-sures were chosen because, first, each is adjusted bythe extent to which a valid response (i.e., observedagreement between homeowner report and inspec-tion record) could be expected purely by chance;and second, as measures of agreement rather thanassociation (e.g., Chi-square), they explicitly distin-guish between patterns of agreement. For example,a valid assessment by the homeowner can eitherreflect that a repair is needed or that a repair is notneeded. Similarly, an invalid assessment may indi-cate that a homeowner did not report a repair need;or conversely, a homeowner reported an unneces-sary repair. Such distinctions are important: Theremay be greater concern if elderly homeowners can-not report needed repairs than if they report unnec-essary repairs. On the assumption that the inabilityto validly report needed repairs reflects the inabilityto recognize that repairs are needed (rather thanembarrassment in the interview situation or the ten-dency to give desirable responses), response errorsin some items (e.g., broken steps, electrical prob-lems) may represent a threat to health and safety.Carried one step further, because recognizingneeded repairs is a prerequisite for undertakingthem, such response errors imply that needed re-pairs are unlikely to be undertaken. This is trouble-some because continued upkeep and repair of thehome is one way in which older homeowners canmaintain their ability to live independently in thecommunity. From an economic perspective, homerepair activity is important as a means of maximizingthe rate of return on the housing unit and preventingneighborhood decline.

The Coleman-Light measure represents the condi-tional agreement that rater B places an item in cate-gory i, given that the item belongs in category i. Inthe present case, rater B is the older homeowner andthe housing inspection record for the housing unitestablishes the actual state of repair need. Two mea-sures of conditional agreement were calculated. Thefirst, K(0), measures the extent to which homeown-

310 The Gerontologist

at University of W

indsor on October 29, 2014

http://gerontologist.oxfordjournals.org/D

ownloaded from

ers make an accurate judgment that a repair is notneeded. The second, K(1), measures the extent towhich homeowners accurately report that a repair isneeded. These measures, therefore; are used to dis-tinguish between two types of valid reporting.Cohen's Kappa, in turn, is a weighted sum of theColeman-Light measures of conditional agreementfor any given housing item and thus provides anoverall measure of the validity of homeowner re-sponses. Further, because Kappa is a weighted sumof the Coleman-Light measures (K(0) and K(1)), val-ues for Kappa should fall between those for K(0) andK(1) for the same item (Liebtrau, 1983).

Both the Coleman-Light measure of agreementand Cohen's Kappa are adjusted by the proportion oftimes that homeowners and inspection reports canbe expected to correspond purely by chance. (Asnoted previously, traditional measures of associationgenerally do not make this adjustment.) Both mea-sures have an upper limit of +1.0, which occurswhen there is perfect agreement. If the classifica-tions are independent, the value will be zero. Deter-mining the lower limit for both measures is morecomplex. In the case of Kappa, Cohen notes thatnegative scores are of little interest (Cohen, 1960).For Coleman-Light the lower limit depends on themarginal distribution.

Results

As shown in Table 1, there are high levels of valid-ity in older homeowners' assessments of the repairneeds in their homes. As noted in Column 1, corre-spondence between homeowner and inspection re-ports occurred at least 70% of the time for 19 of the 36repair categories. The proportion dropped to lessthan 50% for only six items. Further analysis, how-ever, indicated that although the observed propor-tions of valid reports were high, most of this ob-served agreement was attributable to chance. Inaddition, elderly homeowners were particularly un-likely to report accurately that a repair was needed,which is the conditional agreement that is of greatestpolicy importance. Based on Column 5, this type ofdisagreement was generated by the majority of re-pairs. Included are such items as repairs to broken,loose steps, cracked cement in sidewalk, and repairsto light fixtures.

The observed proportions of valid responses sug-gested that elderly homeowners are accurate re-porters of the repair needs of their residences. Fur-ther analysis, however, tempers this sanguineconclusion. Cohen's Kappa indicated that most ofthe homeowners' valid reports were attributed tochance. As noted in Column 2, although 21 repaircategories generated K values greater than zero,most were less than .05, and only four items provedto be statistically greater than zero. Consequently,despite the high levels of observed accuracy in re-porting by older homeowners, the Kappa resultsdemonstrated that after adjusting for chance, thelevel of correspondence between homeowners' re-ports and housing inspection records is only margin-

ally greater than zero for most housing items. Itshould be noted that although the underlying mar-ginal distributions are highly skewed, the over-whelming difference between the observed and thechance-corrected results suggested that this conclu-sion would be unlikely to change if the marginalswere more normally distributed.

Unfortunately, these results reflected general va-lidity and failed to distinguish between the type ofcorrect assessments being made. In particular, arehomeowners correctly identifying items in need ofrepair, or items not in need of repair? Therefore,although the Kappa results are important, theyshowed no distinction between the homeowner'sability to report about needed repairs versus repairsthat are not needed.

The Coleman-Light measure of conditional agree-ment addresses this issue. Two conditions werestudied; first, when the housing item was not in needof repair, K(0); and second, when a repair wasneeded, K(1). Findings regarding the validity ofhomeowner repair reports in these two circum-stances are presented in Columns 4 and 6 of Table 1.For the first condition, K(0) was greater than zero for21 of the 36 housing items and ranged from .1086(repair exterior surface) to 1.000 (grout, caulk wall-joints, and repair glass in door). Of the 21, 18 werefound to be significantly different from zero. Fur-thermore, in most instances K(0) exceeded K, whichsuggested that the homeowner's ability to validlyassess that no repair was needed was greater thanthat implied by the overall chance-corrected mea-sure (K).

Of particular interest are the results under thesecond condition, K(1): whether the homeownervalidly assesses that a repair is needed. Twenty-oneof the entire set of housing items under analysis hadK(1) greater than zero, of which 18 were statisticallysignificant. In contrast to the magnitudes exhibitedby K(0), however, the positive values for K(1) wereminimal and ranged from .0024 (repair interior, bro-ken loose steps) to .1423 (repair electrical outlets).Further, only two housing items (repair porch andrepair electrical outlets) had a conditional agreementgreater than .1000.

The Coleman-Light results added significantly tothe understanding of the ability of elderly homeown-ers to report about the repair needs in their housingunits. Not only are most of the valid assessmentsmade by homeowners largely attributed to chance,but even after adjusting for chance, the homeownerdid not appear to accurately assess items in need ofrepair.

Conclusion

Measurement problems critically affect the validityof survey results. Measures that produce systematicerrors or bias prevent accurate estimates of truepopulation values and increase the likelihood ofdrawing erroneous conclusions. In the present anal-ysis, the validity of elderly homeowners' assessmentsof repair needs for a range of features in their hous-

Vol. 27, No. 3,1987 311

at University of W

indsor on October 29, 2014

http://gerontologist.oxfordjournals.org/D

ownloaded from

Table 1 . Observed Agreement Regarding Repair Need Between Homeowner and Expert With Repair Needs Ranked According toPercentage of Overall Owner/Expert Agreement'(n = 289 to 619 depending on the housing item)

Repair Need

Install door weatherstrippingRepair bath, shower hardwareCaulk, grout toiletRepair masonry wall supportRepair electrical outletsRepair toilet flush mechanismRepair glass in doorRepair projection below roof levelRepair exterior handrailsRepair window screensRepair outside trim, guttersRepair light fixtureRepair porchRepair exterior broken, loose stepsRepair gateRepair interior broken, loose stepsReglass windowsCaulk doorRepair window locks, chainsPaint outside trim, guttersRepair exterior surfaceGrout, caulk exterior stepsPaint exterior handrailsPaint exterior surfaceGrout, caulk walljointsRepair floor tilesCaulk, putty windowsRepair doorPaint projections below roof levelRepair window frame, sash, sillRepair wall wallpaper, panelingRepair ceiling wallpaper, panelingRepair cracked cement in sidewalkPaint doorPaint windowRepair door hooks, bells

Overall agreement

Owner/ExpertAgreement

% Total(D

99.0%97.596.495.993.291.790.189.188.888.487.586.786.484.383.581.977.572.971.769.167.866.365.963.463.262.362.057.753.552.746.846.244.142.733.513.6

K(2)

-.0046- .0037- .0091

0.2016**.1411*.0341

-.0127- .0053-.0025-.0213

.0471

.1794**

.0970**

.0550

.0047

.0222-.0047

.0083

.0057

.0122

.0081- .0020

.0838**

.0177-.0164

.0474-.0112

.0260

.0252

.0204

.0292- .0077- .0032

.0103- .0009

Agreement that:

Repair is not needed

% Owner/ExpertAgreement

(3)

99.6%99.899.5

100.98.398.8

100.99.399.799.998.798.799.197.699.899.399.199.799.897.097.699.798.593.0

100.98.197.598.299.799.797.999.596.499.399.199.1

K(0)(4)

- .0065- .0233-.0315

0.3457**.3840**

1.0000**-.1141-.1221-.1297-.1286

.2223**

.6651**

.2898**

.7221**

.0660

.2648*- .3673

.4020*

.0402

.1086

.4333*-.0354

.2773**1.0000**

- .3270.3826**

-.2847.8182**.7761**.3705**.7746**

-.1542- .4606

.5416*-.5147

Repair needed

% Owner/ExpertAgreement

(5)

0%000

16.510.4

1.900004.3

12.79.13.61.02.400.83.43.30.91.3

14.11.40.66.50.82.72.84.33.82.70.32.50.5

K(1)(6)

.0035

.0201

.00530

.1423**

.0864**

.0173**

.0674

.0027

.0013

.0116

.0263**

.1037**

.0582**

.0286**

.0024

.0116*

.0024

.0042*

.0030

.0064

.0041*

.0010

.0493**

.0090**

.0084

.0253**

.0057

.0132**

.0128**

.0105**

.0149**

.0039

.0016

.0052**

.0005

aFor each repair need listed, Column 1 represents the percentage of all cases where an owner and expert agreed that a "repair isneeded" or a "repair is not needed". For example, taking the first repair need listed "install door weatherstripping", Column 1 indicatesthat owners and experts agreed 99.0 percent of the time that a "repair is needed" or that a "repair is not needed". Column 3 summarizesthe percentage of homeowners who agreed with the expert's report that a repair was not needed. Column 5 presents the percentage ofhomeowners who agreed with the expert's report that a repair was needed. In both instances, the percentage of agreement could rangefrom 0 percent to 100 percent. For example, of owners identified as not needing the installation of door weatherstripping, 99.6 percentagreed with the assessment, while 0 percent of those reported to need door weatherstripping were in agreement. These descriptivemeasures are complemented by Cohen's Kappa, K, in Column 2; the Coleman-Light measure of conditional agreement, K(0), that a repairis not needed in Column 4, and the Coleman-Light measure of conditional agreement, K(1), that a repair is needed in Column 6.

*p<.05; **p<.01.

ing units has been examined. Although the simplepercentages of valid responses were high, more de-tailed statistical testing revealed the need to moder-ate this positive result.

The results provided by Cohen's Kappa, a measureof agreement, highlighted the sensitivity of the ob-served results to chance. Consequently, conclusionsbased solely on raw percentages and not correctedfor chance may overestimate the ability of olderhomeowners to provide valid assessments of homerepair status. In addition, the Coleman-Light mea-sure of conditional agreement showed that much ofthe chance-corrected agreement reflects valid re-

porting that no repair is needed as opposed to repairis needed. Given the policy importance of accuratereporting about possibly unsafe housing conditions(e.g., broken or loose steps, electrical repairs), theseColeman-Light results are disturbing.

Why the elderly have particular difficulty providingvalid reports when a repair is needed is a matter forspeculation. It may reflect their lack of knowledge ofhousing systems or it may reflect their low expecta-tions regarding housing quality. Distinguishing be-tween these (or other) interpretations requires sepa-rate analysis that will identify those factors (e.g.,socioeconomic characteristics of the elderly respon-

312 The Gerontologist

at University of W

indsor on October 29, 2014

http://gerontologist.oxfordjournals.org/D

ownloaded from

dent) underlying the variations in reports of repairneed. Regardless of the cause, the results are sober-ing for the survey designer interested in assessingthe elderly's housing quality. If the older home-owner is not reporting about needed repairs due to alack of awareness, then concerns about the elderly'ssafety and welfare appear warranted. Unfortunately,the extent of this problem cannot be estimated be-cause it is not known whether those who do notreport repair needs represent a specific subset ofelderly respondents or whether they represent all, oralmost all, respondents.

If only a subset of respondents do not report repairneeds, it may be necessary to redesign the samplingprocess to exclude them, or to redesign the ques-tionnaire to elicit more accurate responses (Herzog& Rodgers, 1982). Alternatively, if all or almost allrespondents have this problem, then there will be asystematic bias in the estimation of the overall meanrate of repair need which should be adjusted.

References

Botwinick, J. (1978). Aging and behavior (2nd ed.). New York: Springer.Cannell, C , Fisher, C , & Bakker, T. (1965). Reporting of hospitalization in

the health interview survey. Health Statistics, Series 2, No. 8. Washing-ton, DC: U. S. Department of Health, Education and Welfare.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educationaland Psychological Measurement, 20, 37-46.

Comptroller General. (1977). The well-being of older people in Cleveland,Ohio. Washington, DC: U. S. General Accounting Office.

Craik, F. I. M. (1977). Age differences in human memory. In J. E. Birren &K. W. Schaie (Eds.), Handbook of the psychology of aging. New York:Van Nostrand Reinhold.

Davis, J. (1971). Elementary survey analysis. Englewood Cliffs, N. J.: Pren-tice-Hall.

Duncan, G. Jr., & Hill, D. H. (1985). An investigation of the extent andconsequences of measurement error in labor economic survey data.Journal of Labor Economics, 3, 503—532.

Ferber, R. (1955). Item nonresponse in consumer surveys. Public OpinionQuarterly, 11, 399-415.

Gergen, K.)., & Back, K. W. (1966). Communication in the interview and thedisengaged respondent. Public Opinion Quarterly, 33, 17-33.

Groves, R., & Magilavy, L. (1980). Estimates of interviewer variance intelephone surveys. Proceedings of the 1980 American Statistical Associa-tion Survey Research Method Section. Washington, DC: AmericanStatistical Association.

Herzog, A. R. (1979). Attitude change in older age: An experimental study.Journal of Gerontology, 34, 697-703.

Herzog, T. N., & Rodgers, W. (1982). Surveys of older Americans: Somemethodological investigations. Final Report to the National Institute onAging, Ann Arbor, M l : Institute for Social Research.

Kain, J. F., & Quigley, J. M. (1972). Note on owner's estimate of housingvalues. Journal of the American Statistical Association, 67, 803-807.

Klein, R. L. (1972). Age, sex, and task difficulty as predictors of socialconformity. Journal of Gerontology, 27, 229-236.

Klein, R. L, & Birren, J. E. (1972). Age differences in social conformity on atask of auditory signal detection. Proceedings of the 80th Annual Con-vention, American Psychological Association. Washington, DC: Ameri-can Psychological Association.

Kish, L , & Lansing, J. B. (1954). Response error in estimating the value ofhomes. Journal of the American Statistical Association, 49, 520-538.

Lawton, M. P. (1980). Social and medical services in housing for the aged.Washington, DC: U. S. Department of Health and Human Services.

Liebetrau, Albert, M. (1983). Measures of association. Sage University PaperSeries on Quantitative Application in the Social Sciences, 07-036. Bev-erly Hills and London: Sage Publications.

Marquis, Kent (1981). Record check validity of survey responses: A reassess-ment of bias in reports of hospitalizations. Santa Monica, CA: RandCorporation.

National Council On The Aging. (1975). The myth and reality of aging inAmerica. Washington DC: National Council on the Aging.

Neter, J. (1970). Measurement error in reports of consumer expenditures.Journal of Marketing Research, 11-25.

Newman, S. J., Curtin, R., & Chen, A. (1981). Home repair services for theelderly: An evaluation of Baltimore's home maintenance program (PhaseOne). Final Report to U. S. Department of Housing and Urban Develop-ment, Office of Policy Development and Research. Ann Arbor, M l :Institute for Social Research.

Palmer, G. L. (1943). Factors in the variability of response in enumerativestudies. Journal of the American Statistical Association, 38, 143-152.

Rabushka, J., & Jacobs, A. (1980). Old folks at home. New York: The FreePress.

Sudman,S., &Bradburn, N. M. (1974). Response effects in surveys: A reviewand synthesis. Chicago: Aldine.

U. S. Department of Commerce, (various years). Current housing reports,annual housing survey. Washington, DC: U. S. Government PrintingOffice.

U. S. Department Of Housing and Urban Development. (1979). How wellare we housed? 4. The Elderly. Washington, DC: U. S. Department ofHousing and Urban Development.

Vol. 27, No. 3,1987 313

at University of W

indsor on October 29, 2014

http://gerontologist.oxfordjournals.org/D

ownloaded from