bias resulting from missinginformation: some epidemiological

British Journal ofPreventive and Social Medicine, 1977, 31, 131-136

Bias resulting from missing information:Some epidemiological findingsANTONY COX, MICHAEL RUTTER, BRIDGET YULE, AND DAVID QUINTON

From the Department of Child and Adolescent Psychiatry, London

sumUmARY The biases resulting from missing information were examined in three psychiatricepidemiological studies. In each study, cases with missing information could be compared with themain sample because data were available from several sources or at several points in timethrough a longitudinal study. In almost all instances, cases with missing data differed systemati-cally in terms of variables crucial to the questions being studied. In general, they tended toinclude a higher proportion with problems of various kinds-such as, behavioural deviance,reading backwardness, child or adult psychiatric disorder, and marital discord. The characteristicsor circumstances of those giving information were generally more strongly associated withco-operation in testing or interviewing than the characteristics of those about whom informationwas sought. In some situations, the nature and degree of distortion resulting from missinginformation could lead to biased results.

In almost all studies data are lacking from aproportion of the sample and it is sometimesassumed that the missing cases have not distortedthe findings. When the matter has been examined,this assumption has often received some supportfrom the finding that the cases missed did notdiffer from the remainder of the sample in terms ofsuch characteristics as age, sex, and social class.However, just because they have been missed, it hasrarely been possible to determine if the missed casesdiffered in terms of the variables central to thehypotheses being tested or the disorders beingstudied. Such information is likely to be availableonly when information is sought from severalsources or on several occasions such as during thecourse of a longitudinal study.The findings in these situations are much less

reassuring. Thus, in the Isle of Wight study (Rutteret al., 1970) children whose parents failed tocomplete behavioural questionnaires were twiceas likely as other children to receive a deviant scoreon questionnaires completed by teachers. Similarly,West and Farrington (1973) found that boys whoseparents refused to be interviewed were more likelyto become delinquent than boys whose parentsco-operated with the study. In the survey byShepherd et al. (1971) the children whose parentsfailed to complete questionnaires were more likelyto have low attainments, to show behaviouraldifficulties in class, and to become delinquent later.

In her 30-year follow-up study of psychiatric clinicattenders, Robins (1963) discovered relatively fewfactors associated with refusal to be interviewed,but sociopathic individuals were more difficult totrace because of their greater mobility both fromhouse to house and also from area to area (Robins,1966). Sims (1973) noted that patients withneurosis who were difficult to trace in a follow-upstudy were more likely to have died.The opportunity to examine more closely the

possible biases resulting from missing informationarose from three interrelated psychiatric epidemio-logical studies. Data were available from severalsources in all these studies and in two, longitudinaldata were also obtained. This paper summarisesthe main findings.

Samples and methods

INNER LONDON STUDYIn 1970 the total population of 10-year-old childrenattending local authority schools in an innerLondon borough was assessed by group tests ofreading and non-verbal intelligence and by aquestionnaire on behaviour completed by teachers(Berger et al., 1975; Rutter et al., 1975). From thissurvey two subgroups (a random sample and asample of children with deviant scores on theteacher's questionnaire) were chosen for moreintensive studies by means of interviews with

131

Protected by copyright.

on Novem

ber 19, 2021 by guest.http://jech.bm

j.com/

Br J P

rev Soc M

ed: first published as 10.1136/jech.31.2.131 on 1 June 1977. Dow

nloaded from

http://jech.bmj.com/

Antony Cox, Michael Rutter, Bridget Yule, and David Quinton

parents and teachers.Three years, and again four years later, the

same total population of children (then at secondaryschool) was re-assessed by means of group testsof reading and of non-verbal intelligence and by abehavioural questionnaire completed by teachers(Rutter, 1977). Special attention was paid to thechildren at the 25 secondary schools in, orimmediately adjacent to, the borough studied in1970.

ISLE OF WIGHT STUDYIn 1969 the total population of 10-year-old childrenliving on the Isle of Wight and attending localauthority schools was assessed in a somewhatsimilar way to that in the inner London study(Berger et al., 1975; Rutter et al., 1975). Twosimilarly defined subgroups were studied intensivelyby interviewing both the parents and teachers.

FAMILY ILLNESS STUDYA prospective study of families in which oneparent had a psychiatric disorder was carried outin the same inner London borough as that used forthe general survey of 10-year-old children. Thesample consisted of all patients who spokecolloquial English at home, who had at least onechild under the age of 15 years, and who hadbeen newly referred to a psychiatric clinic duringthe course of a 10-month period (Rutter, 1970).At the time of initial referral, and again one andfour years later, detailed systematic interviews wereheld with the patient and spouse (if there was one) toobtain information on patterns of family life, themarriage relationship, and on the psychiatricstate of all family members (Rutter and Brown,1966; Quinton et al., 1976). Shorter interviews werecarried out two and three years after the initialreferral.

Behavioural questionnaires were completed byteachers for all children of school age in the familiesat the time of initial referral and again at each ofthe four years of follow-up. Questionnaires werecompleted in the same way throughout the follow-up study for a control group of children of the sameage, sex, ethnic background, and in the same schoolclass as the patients' children at the start of thestudy.

Results

GROUP TESTSIn the inner London study, reading tests wereadministered to all 13-year-old children attending24 secondary schools* serving one borough. TestsIn later stages of the study data were obtained for 25 schools butat this stage one school failed to carry out testing.

were repeated to include children absent on the dayof the first tests. However, even after severaltesting occasions 8 6% of the children had stillnot been tested. Because the children had previouslybeen studied in primary school, it was possible touse measures obtained at 10 years of age tocompare in the 13-year-old children the absenteeswith the children tested at that age. This is done inTable 1.

Table 1 Characteristics of children with missing dataon group tests (reading test 1973)

Reading test at 13 years

Characteristics at 10 years Children Childrenabsent tested(n = 136) (n = 1448)

Behavioural deviance on teacher'squestionnaire 37*5% 20*8Y%**Reading at least 2 SD below mean 23-5% 14.-0%**

**P <0*01

Of the children successfully re-tested at the ageof 13, 14% when aged 10 years had had a readingscore on a group test at least two standarddeviations below the standardised sample mean.But, among the absentees at the age of 13, 23-5%had had a similarly low reading score at age 10years. This difference is significant at 1% level.In the same way, nearly twice as many of theabsentees had had deviant scores on the teacher'squestionnaire at the age of 10 compared withthose tested at 13 years (37*5% v. 20 8%;P <0-01).

Table 2 Characteristics of children with missing dataon group tests (reading test 1974)

Reading test at 14 years

Characteristics at 10 years Children Childrenabsent tested(n = 116) (n = 1559)

Behavioural deviance on teacher'squestionnaire 40 5% 14-9Y/.**Reading at least 2 SD below mean 24-8% 13-6%**

*P <0.01

The same reading test was given again a yearlater, in 1974, because one school had not giventhe reading test and it was hoped that theproportion of children with missing informationmight be reduced. This time the absenteeism ratewas 6-9%. These results are shown in Table 2.Once more, twice as many of the absentees had beenbehaviourally deviant at 10 years of age comparedwith those present at 13 (40 5% v. 14-9%;P <0'01).

132


on Novem


j.com/

Br J P

rev Soc M


nloaded from


Bias resulting from missing information: Some epidemiologicalfindings

Table 3 Characteristics of children with missing datain group tests (non-verbal IQ test 1973/74)

IQ test at 13 or 14 years

Characteristics at 10 years absent Childrnboth years(n = 44) (n = 1631)

Behavioural deviance on teacher'squestionnaire 40*9y% 22 -1%**Reading at least 2 SD below mean 29-3%, 14 0%**

*-P <0-01

The non-verbal intelligence test was given in1974 to children who had repeatedly missed thetest in 1973. In spite of giving the test on severaldifferent days in 1974, there were still 44 children(2 6%) who were absent on all testing occasionsin both years. Table 3 shows the characteristicsof these persistent absentees at 10 years of agecompared with the rest of the sample. Again the ratesof behavioural deviance (40 9% v. 22 1%; P <0 01)and poor reading attainment (29-3% v. 140%;P <0'01) were twice as high among the absentees.

PARENTAL INTERVIEWSIn the Isle of Wight study 202 children wereselected for study by parental interview. Interviewswere not obtained in 16 cases (7'9%); 12 familiesrefused to participate and four could not be traced.In the inner London study there were no parentalinterviews for 21 (8 4%) out of 249 selectedchildren (20 refusals and one not traced).

Table 4 Characteristics of children whose parents werenot interviewed (Isle of Wight and London studiescombined)

Parents not ParentsCharacteristics of children intervtewed Interviewed

(n = 37) (n = 414)Behavioural deviance on teacher'squestionnairet 757%/ 58 -2 /*Mean non-verbal IQ (London only) 86-9 86-9Mean reading quotient (London only) 85-8 87'9

tThe samples interviewed were deliberately stratified so that somethree-fifths of the children would be deviant on the teacher'squestionnaire. Accordingly, the proportions do not represent popula-tion prevalence figures. For prevalence data see Rutter et al., 1975.*P<0.05

In both samples a higher proportion of thechildren whose parents were not interviewed wererated as showing behavioural deviance on theteacher's questionnaire (P <0 '05). However, thisdifference was accounted for by the parents of boys.Of the 178 boys who were behaviourally devianton the teacher's questionnaire, 17 had parentswho were not interviewed compared with only oneout of the 94 non-deviant boys. The total groups ofchildren whose parents were not interviewed did

not differ in terms of sex, non-verbal IQ, orreading attainment.The above data apply to differences between

parents interviewed and not interviewed. The innerLondon study data allow similar comparisons withrespect to degree of co-operation. Although veryfew families had no interviews at all there was alarger proportion whose interviews were incompletein some respect. Among the 190 families in whichthe key child was living with both parents therewere 43 for whom data were obtained from onlyone informant*. In an additional eight cases theinterview could not be completely finished, usuallybecause the husband would not let it continue.The differences between these 51 families withincomplete information and the 139 with completedata are summarised in Table 5.

Table 5 Characteristics of families with missing dataon parental interview (inner London borough study)

Incomplte CompketeCharacteristics offamily interviews Interviews

(n = 51) (n = 139)

Child has conductlmixed disorder(interview rating) 23 *55. 7 * 2Markedmaritaldiscord 38-1% 16.6%**Non-manual social class 18 0% 20-4%Psychiatric disorderof father 152% 124%of mother 36*2;/% 27*5%.

Child has behavioural deviance onteachers questionnairet 64-7 559 *7%tSee footnote to Table 4***P <0 001**P <0X01

Incomplete interviews were not related to the sexof the child, the social class of the family, thepsychiatric state of the parents, or behaviouraldeviance in the child as assessed by teacher'squestionnaire. However, children in families withincomplete interviews were more likely to have aconduct or mixed emotional/conduct disorder asrated from parental interview (25'5% v. 7-2%;P <0'001). Marital discord was also twice ascommon (38'1% v. 16'6%; P<0'01) if theinterviews were incomplete. Both the psychiatricrating (Rutter et al., 1975) and the marital rating(Quinton et aL, 1976) have been shown to bereliable and valid.

Broadly similar findings were obtained by com-paring mothers rated by the interviewer as fullyco-operative and those rated as showing some degreeof reluctance or non-cooperation. Less than fullyco-operative mothers were more likely to have a childwith a conduct or mixed emotional/conduct

*In this study the interview with the husband was mainly used as acheck on information provided by the wife, rather than as a primarysource of data.

133


on Novem


j.com/

Br J P

rev Soc M


nloaded from



Table 6 Family characteristics and degree of informantco-operation (inner London borough study)

Informants

Characteristics offamily Any degree Fullyof reluctance co-operative(n = 85) (n = 142)

Child has conduct/mixed disorder(interview rating) 20-0% 7-8%,**Marked marital discord 30 3% 17*2 %**Father has psychiatric disorder(interview rating) 26-5Y% 12 * I %**Child has behavioural deviance onteacher's questionnairet 63*5 /, 656%

tSee footnote to Table 4**P <0-01

disorder (200% v. 7 - 8 %; P <0-01), more likelyto have a discordant marriage (30 3% v. 17.2%;P <001), and more likely to have a husband withsome kind of psychiatric disorder (26*5% v.12.1 %; P <0 01).Various aspects of co-operativeness could also be

assessed in the family illness study in whichinterviews with the husband and wife were of equalimportance and were dealt with similarly. At thetime of the first family contact one or other was anewly referred psychiatric patient. The only twovariables associated with co-operativeness werepatient status and sex of informant.

Thus, in the initial contact with the family (incases when the husband and wife were livingtogether), fewer interviews were obtained withspouses than with patients (85 % v. 98%; P <0-001),and there was a tendency for men to co-operateless well than women (88% v. 95%; P <O01;patients and spouse combined). The poorestco-operation was by male spouses (79 * 2%) andthe best by female patients (99 0%). During thecourse of the four-year longitudinal study thepattern remained much the same.Thus in the first and fourth years of follow-up,

spouses were twice as likely as patients to refuseinterview or to be repeatedly out (first year, 18% v.9%; P <005; fourth year, 23% v. 9%, P <001).Similarly, twice as many men could not be inter-viewed (first year, 19% v. 7%, P <0-01; fourthyear, 21 % v. 8%, P <001). Among the malespouses it was those who were completely wellor those with personality disorders (rather thanthose with acute transient psychiatric problems) whowere most likely to be uncooperative (P <0 025).

This longitudinal study also made it possible toobserve the extent to which non-cooperation wasa persistent family characteristic. There weremany examples of people who refused interview onone occasion but yet agreed on another. Further-more, there were very few families in which bothhusband and wife persistently refused interview.

Nevertheless, when at least one member of acohabiting couple refused to be interviewed on thefirst occasion, there was an increased chance that oneor both would refuse to be interviewed subsequently.When both husband and wife were interviewed atthe outset, both were interviewed in the fourthyear of follow-up in 72% of cases in which theycontinued to live together. In contrast, if only onehad been interviewed at the outset, in only 33%of cases were both co-operative in the final year(P <0.01).

MOVE OF HOUSEIn longitudinal studies a further source of missinginformation is migration of the family to otherparts of the country (or abroad). The familyillness study afforded the opportunity to determinewhether the migrating families differed from thoseremaining in London. During the course of thefour-year follow-up, 13.5% of the patients'families and 10'6% of the controls moved out ofinner London. The very small difference betweenthese two groups indicates that parental psychiatricstate is unrelated to the likelihood of internalmigration. Within the patient sample migration wasalso unassociated with the presence of maritaldiscord (37.9% of the migrants showed discordat the initial assessment compared with 44-8% ofthe non-migrants). Furthermore, within the controlgroup behavioural deviance in the children at thestart of the study was unrelated to migrationout of London during the next four years (13 6%of the migrant children were behaviourally deviantcompared with 18 * 6% of the non-migrant children).

Discussion

In all the epidemiological studies we haveconsidered there have been substantial and importantdifferences between individuals who co-operatedwith interviews or tests and those who did not. Thefindings need to be discussed from two ratherdifferent perspectives. Firstly, what light is thrownon the process of co-operation or non-cooperationin research projects? Secondly, what implicationsfor epidemiological research stem from the natureand degree of bias resulting from missing informa-tion?

In keeping with previous studies, cases withmissing data were not usually distinguishable interms of demographic characteristics such as socialclass. On the other hand, in almost all instances thechildren or families with missing information diddiffer in terms of the variables crucial to thequestions being studied. In general, the missingcases included a higher proportion who were

134


on Novem


j.com/

Br J P

rev Soc M


nloaded from


Bias resulting from missing information: Some epidemiologicalfindings

deviant or disordered in some way. However, thepattern was not the same in all situations.

It appears that co-operation or willingness toparticipate in a study is best understood inrelationship to the characteristics or circumstances ofthe person being interviewed or tested. Onlyindirectly is co-operation related to the character-istics of the individuals about whom information isbeing sought.

Thirteen or 14-year-old children who wererepeatedly absent from school at the time testswere administered were twice as likely as otherchildren to have been behaviourally deviant and/orbackward in reading when they were younger.Persistent truancy is more common in boys whoare delinquent or who have low scholasticattainments, and boys who have poor educationalprogress and bad conduct at 10 years of age arequite likely to have the same difficulties at 14(Graham and Rutter, 1973; Rutter, 1977). Labouvieet al. (1974) also found that adolescents whorefused to participate or who were unavailable fortesting had lower scores on cognitive tests thanthose who had co-operated.Conduct disorders in childhood and adolescence

are linked with school truancy and with personalitydisorders in adult life (Robins, 1966), so it is to beexpected that childhood antisocial problems wouldtend to predict co-operation in follow-up studies.Thus Bebbington (1970), in a 10-year follow-upstudy of London adolescents, found that those whostalled or who refused to be interviewed at follow-uptended to have been of lower intelligence whenthey were aged 13. These people were also morelikely to have been delinquent.

In the general population studies, husbands wereless likely to be interviewed if their wives gaveevidence that the marriage was unhappy. In someinstances the women dissuaded the interviewersfrom approaching their partners. The lack ofco-operation by the husbands in discordantmarriages could have been either because of theirreaction when they found out that their wives haddisclosed family difficulties or because wivesblocked the contact with the interviewer for fear ofthat reaction. The reduced co-operation in womenwho had husbands with a psychiatric problem mayhave been based on similar fears. On the other handthe children whose parents could not be intervieweddid not differ from other children in IQ orreading attainment. The difference in the proportionof those with conduct difficulties was presumablybecause of family discord and parental problems(Rutter, 1971).

Co-operation was better and not so clearlyrelated to family or personal problems in the

longitudinal investigation of patients' families.This is probably a consequence of several factors.Firstly, strenuous efforts were made by the projectteam to maintain contact and to obtain interviews.Secondly, there was such a high rate of parentaldiscord and psychiatric disorder in the group as awhole that it was less likely that these factorswould distinguish individuals who co-operatedless well. Thirdly, they constituted a group who hadrecently attended a psychiatric service and mighttherefore be more inclined to co-operate withworkers who were linked with the services whichhad helped them at their time of need. There issupport for this interpretation in the finding thatpatients remained more co-operative than theirspouses throughout the four years of the study.

In all the current studies men proved moredifficult to interview than women. In part this mayhave been because the interviews were primarilyfocused on children and men may not have seenchildren as their province. In part, too, it mayhave been that practical difficulties stemming fromlong working hours may have prevented interviewswith men who would have been willing to be seenhad time permitted. However, it is unlikely thatthis entirely explains the difficulty as other studieshave noted that fathers were crucial in determiningco-operation in conjoint family treatment (Shapiroand Budman, 1973; Slipp et al., 1974).Our findings, as well as those of others

(Bebbington, 1970; West and Farrington, 1973)indicate that uncooperative informants are not ahomogeneous group. West and Farrington suggestthat uncooperative parents tend to be of one of twotypes: either those with serious social or psychologi-cal difficulties or alternatively those without grossdisturbance but who are overprotective, self-contained, and independent. Our findings suggestthat men with a personality disorder were mostlikely to be uncooperative. Those with acutedisorders and those who had sought psychiatrichelp, on the other hand, were usually verywilling to be interviewed. Perhaps some mentaldisorders are associated with a great willingness todiscuss personal matters whereas with others theopposite is true (Cozby, 1973). Lack of co-operationmay reflect antisocial attitudes and disturbedpersonal relationships or it may stem from a dislikeof self-disclosure or intimacy which has nothingto do with psychiatric disorder.

In the current study, internal migration was notassociated with any measure of disorder in parentsor children. This was also so in the Bebbington(1970) study, but not in Robins's investigation(Robins, 1966). The matter requires further studyand it would not be justified on these findings to

135


on Novem


j.com/

Br J P

rev Soc M


nloaded from



assume that migration is never a source of 'bias'.It is necessary to consider the implications for

epidemiological research which stem from thefindings of bias resulting from missing infor-mation. Our evidence, as well as that from previousstudies, clearly shows that individuals or familieswho are not interviewed or tested, or for whom dataare partially lacking, tend to differ in importantways from the main samples with complete data.Frequently cases with missing data included ahigher proportion with problems of various kinds.

Obviously, the nature of this bias could causeserious distortions in results, but the extent towhich results will in fact be distorted depends ona variety of other considerations. In all the studieswe undertook, persistent efforts by the investi-gators led to a very high level of co-operationso that refusal rates were kept below and usually wellbelow 10%. In these circumstances the distortionof total population rates which results frommissing data is trivial. Moreover, because datawere always available from multiple sources itwas possible to allow for the missing informationif this was required.However, there are two circumstances in which

missing information may cause more seriousdistortions. The first is when conclusions are beingdrawn from deviant subsamples rather than fromthe population as a whole. This may be illustratedby reference to Table 2. Only 7% of childrenmissed the reading test at 14 years. Although this7% was nearly twice as likely to be backward inreading, the missing data will make very littledifference to the population mean scores onreading, simply because 7% is such a smallproportion. On the other hand, if the studyconcerned the reading attainments of behaviourallydeviant children the distortion would be muchgreater because 17% of behaviourally deviantchildren missed the tests.The second circumstance is when refusal rates

are much higher than they were in our investi-gations. It is not at all uncommon to publishepidemiological or longitudinal studies in whichdata are missing for more than one-quarter of thepopulation, and sometimes even for half. In thesecases distortions could be considerably greater.What we can conclude is that it behoves allinvestigators to keep non-cooperation rates as lowas they can and to check as far as possible on thecharacteristics of cases with missing data. Multiplesources of data and repeated tests or interviewsare invaluable for this purpose.Reprints from M. Rutter, Department of Child andAdolescent Psychiatry, Institute of Psychiatry, DeCrespigny Park, Denmark Hill, London SE5 8AF.

References

Bebbington, A. C. (1970). The effect of non-responsesin the sample survey with an example. HumanRelations, 23, 169-180.

Berger, M., Yule, W., and Rutter, M. (1975). Attainmentand adjustment in two geographical areas. II. Theprevalence of specific reading retardation. BritishJournal ofPsychiatry, 126, 510-519.

Cozby, P. C. (1973). Self-disclosure: a literature review.Psychological Bulletin, 79, 73-91.

Graham, P., and Rutter, M. (1973). Psychiatricdisorder in the young adolescent: a follow-up study.Proceedings of the Royal Society of Medicine, 66,1226-1229.

Labouvie, E. Q., Bartsch, T. W., Nesselroade, J. R.,and Baltes, P. B. (1974). On the internal and externalvalidity of sample longitudinal designs. ChildDevelopment, 45, 282-290.

Quinton, D., Rutter, M., and Rowlands, 0. (1976).An evaluation of an interview assessment of marriage.Psychological Medicine, 6, 577-586.

Robins, L. N. (1963). The reluctant responder. PublicOpinion Quarterly, 27, 276-286.

Robins, L. N. (1966). Deviant Children Grown Up.Williams and Wilkins: Baltimore.

Rutter, M. (1970). Sex differences in children's responsesto family stress. In Thte Child in His Family. Editedby C. Koupemik. Wiley: New York.

Rutter, M. (1971). Parent-child separation: psychologi-cal effects on the children. Journal of Child Psychologyand Psychiatry and Allied Disciplines, 12, 233-260.

Rutter, M. (1977). Prospective studies to investigatebehavioural change. In Methods of LongitudinalResearch in Psychopathology. Edited by J. S. Strauss,H. M. Babigian, and M. Roff. Plenum: New York.

Rutter, M., and Brown, G. (1966). The reliability andvalidity of measures of family life and relationshipsin families containing a psychiatric patient. SocialPsychiatry, 1, 38-53.

Rutter, M., Cox, A., Tupling, C., Berger, M., and Yule,W. (1975). Attainment and adjustment in twogeographical areas. I. The prevalence of psychiatricdisorder. British Journal of Psychiatry, 126, 493-509.

Rutter, M., Tizard, J., and Whitmore, K., Editors(1970). Education, Health and Behaviour. Longman:London.

Shapiro, R. J., and Budman, S. H. (1973). Defection,termination and continuation in family and individualtherapy. Family Process, 12, 55-67.

Shepherd, M., Oppenheim, B., and Mitchell, S. (1971).Childhood Behaviour and Mental Health. Universityof London Press: London.

Sims, A. C. P. (1973). Importance of a high tracing ratein long-term medical follow-up studies. Lancet, 2,433-435.

Slipp, S., Ellis, S., and Kressel, K. (1974). Factorsassociated with engagement in family therapy. FamilyProcess, 13, 413-428.

West, D. J., and Farrington, D. P. (1973). WhoBecomes Delinquent? Heinemann: London.

136


on Novem


j.com/

Br J P

rev Soc M


nloaded from


bias resulting from missinginformation: some epidemiological

Documents