DTC Quantitative Methods

Data Sources: Secondary Analysis (& Official Statistics)

Friday 13th January 2012

 

What is ‘secondary analysis’?

• Hakim: “…any further analysis of an existing dataset which presents interpretations, conclusions or knowledge additional to, or different from, those presented in the first report on the inquiry as a whole and its main results”.

• Dale et al.: “secondary analysis implies a re-working of data already analysed”.

• Hyman: “the extraction of knowledge on topics other than those which were the focus of the original surveys”.

Online course extracts: Dale et al. 1988; Dale et al. 2008

Sources for secondary analyses

• Surveys

• The Census

• Administrative and/or public records

• Longitudinal studies

• Qualitative studies

The UK Data Archive (http://www.data-archive.ac.uk) now catalogues data from surveys and qualitative studies, as well as the Census, historical data, international country-level databases, etc.

Some sources specifically geared towards secondary analysis

• The British Social Attitudes Survey• Understanding Society

http://www.understandingsociety.org.uk/• The Timescapes qualitative longitudinal study:

http://www.timescapes.leeds.ac.uk/

Benefits of secondary analysis

• It avoids costs in money and time that would make primary research impractical, especially for a lone researcher.

• It allows one to benefit from the fieldwork expertise of professional organizations.

• Cross-national and historical research become more of a practical possibility.

• Secondary analyses of longitudinal data facilitate studies of change over time.

• Large, nationally representative samples facilitate sophisticated, generalisable analyses, and (sometimes) analyses relating to small/relatively inaccessible minorities.

Sampling error (again!)

There will always be somesampling error

...but with a large sample it one can be more confident that it will proportionally smaller.

The expected extent of sampling error in a sample is expressed in terms of confidence levels

(e.g. that you’re 95% confident of being no more than a stated amount wrong about the proportion of the

population who are Roman Catholic, given how many people in your sample were Roman Catholic)

A population of ten peoplewith $0 - $9

The sampling distribution of samples of size 1

The sampling distribution of samples of size 2

The sampling

distributions of samples

of size3 and 4

Bridging the quantitative/qualitative ‘divide’

• Dale et al. comment that “qualitative research can greatly enhance the value of secondary analysis by providing greater depth of information, particularly by suggesting the underlying processes that are responsible for the observed relationships”.

Why is there a shortfall in secondary analyses in the UK? (particularly in some disciplines, e.g. Sociology)

• A lack of quantitatively-orientated researchers.• The legacy of critiques of quantitative research

methods.• More specifically, the legacy of critiques of

official statistics.• Although it’s now more of a question of inertia

than of ongoing scepticism?

Themes within critiques of official statistics

• Concerns about coverage

• Concerns about measurement

• Epistemological concerns

• ‘Political concerns’

A ‘damning’ quote?

“It’s [i.e. the state’s] economic and political functions are embedded in the production of official statistics, structuring both what data are produced and how this is done... only by understanding that statistics are produced as part of the administration and control of a society organised around exploitative class relations can we grasp their full meaning”

(Miles and Irvine, 1979).

However…• Analyses of official data can produce substantively

interesting results

• The producers and users of official statistics are normally very concerned about the errors in data and the data’s limitations,

• The conceptual issues arising from the use of official statistics are not dissimilar to those arising in other forms of sociological research.

• Analyses of official data have been used to critique governments with respect to issues such as unemployment, health inequalities, etc.

(The first three of the above bullet points are suggestions by Bulmer)

... nevertheless

As Hindess commented:• “Official statistics are never mere givens to be

taken as they are or else dismissed as inadequate. Like other productions they must be explained in terms of the conditions and instruments of their production”.

• “As structured social products they [i.e. official statistics] can [and should be!] be critically assessed”.

Official statistics or official data?

• Published official statistics have justifiably been viewed with some scepticism.

• However, the analysis of official data by a secondary analyst can avoid some of the problems.

• Given access to the ‘raw’ official data, she or he can manipulate them in ways different to how they were processed to produce published official statistics.

Are UK official statistics getting more independent?

• A Statistics Board resulting from the Statistics Bill of July 2007, renamed the UK Statistics Authority in February 2008 (see: http://www.statisticsauthority.gov.uk/) is:

• “... an independent body operating at arm's length from government as a non-ministerial department, directly accountable to Parliament.  … [its] overall objective is to promote and safeguard the quality of official statistics that serve the public good. It is also required to safeguard the comprehensiveness of official statistics”.

Some more specific developments: I

• OPCS [now ONS] Disability Surveys were criticised for not adequately reflecting disabled people’s perspectives on their disabilities.

(see Abberley’s chapter in Levitas and Guy, 1996).

• However, they were nevertheless used for some interesting and useful secondary analyes (see Pole and Lampard, 2002, Ch. 7).

• More recently, the Office for Disability Issues (ODI) brought together a group of disabled people as a reference network, in part to facilitate the effective design of a new longitudinal disability survey, the Life Opportunities Survey (LOS):

(see http://www.ons.gov.uk/about/surveys/a-z-of-surveys/life-opportunities-survey/index.html)

Some more specific developments: II

• A number of UK government surveys now (since 2009) ask a question on sexual orientation, following a question being asked in the 2007 Citizenship Survey (and resulting from ONS's Sexual Identity Project, established in 2006).

• This development reflects more general governmental concerns about the availability of ‘equality data’.

• However, it does not seem that a question will be asked on this topic in the 2001 Census!

• The consensus also seems to be that the results generated by the question will under-estimate non-heterosexual orientations.

“What is the moral? Must have a moral…”

• Whether the source of their data is official or non-official, secondary analysts should gain an extensive knowledge of the research design and data collection process.

• This allows the secondary analyst to adopt an informed and suitably critical approach to their assessment of the validity and value of their data source(s).

The data source for the slide title (I think) is “A Funny Thing Happened on the Way to the Forum” (Sondheim)

Key issues in secondary analysis(according to Dale et al.)

• What was the original purpose of the study and what conceptual framework was used? Who was responsible for collecting the data?

• What data did the study collect and how were variables such as occupational class operationalized?

• What was the sample design that was used and what was the level and pattern of non-response?

Is there documentation available in relation to?

• Sample selection

• Patterns of (non-)response

• Interview schedules [Questionnaires]

• Instructions to interviewers

• The coding of answers

• The construction of derived variables

Some other relevant questions...• Is secondary analysis an appropriate approach given the

researcher’s objectives?

• Does the secondary analyst know the topic area well enough to be able to interpret and evaluate the information available?

• What similarities and differences are there between the conceptual frameworks of the original researchers and of the secondary analyst?

• Are the data recent and extensive enough for the secondary analyst’s purposes?

• How consistent is the information with information from other sources?

• Is the information representativeness enough to support generalisations? Is weighting needed to correct for a lack of representativeness?

An example: the General Household Survey

Advantages of the GHS include• a large sample size• the fact that it has been repeated more or less

annually since the early 1970s, which allows trends to be examined

• a broad agenda which means that relationships between concepts belonging to different policy areas can be examined

• a hierarchical structure, which allows linkages between different members of the same household to be examined (Dale et al.)

Disadvantages of the GHS include

• That it has been cancelled, although some of its components have been reassigned to other, less satisfactory surveys...

Some examples of sources and issues from Richard’s research

• Social Change and Economic Life Initiative (SCELI) main survey (1986)

• National Survey of Sexual Attitudes and Lifestyles II (2000) [and various other couple-related surveys]

• General Household Survey (1991 & 2005)• British Election Study (1987)

See also Pole and Lampard, 2002, Ch. 7.

Reasons for the end of a cohabiting or marital relationship

(as shown on a NATSAL II showcard)• Unfaithfulness or adultery

• Money problems

• Difficulties with our sex life

• Different interests, nothing in common

• Grew apart

• Not having children

• Lack of respect or appreciation

• Domestic violence

• Arguments

• Not sharing household chores enough

• One of us moved because of a change in circumstances

(for example, changed jobs)

• Death of partner

• Another reason (please say what)

...and the categories that had to be added

• Drink, drugs or gambling problem• Mental health or related problem• Problem with children/step-children• Never at home (e.g. always out with friends)• Problems with parents/in-laws/family• Age-related problems (e.g. big age difference)• Another relationship involved• Lived in/moved to a different country/area• Still in relationship, but stopped living together• Change of mind/feelings/personality• Partner just left without any explanation

Weighting: A commonly-needed adjustment

• This is used when a particular group has been ‘over-sampled’ (or ‘under-sampled’). This occurs in ‘disproportionate sampling’.

• It assigns some cases more weight than others on the basis of the different probabilities of selection each case had.

• The appropriate approach is to give each case a weight that is (proportional to) the inverse of that case’s selection probability.

Weighting example• I have a population of 10,000 university students that includes 10%

minority ethnic students. • I want to sample 100 people and to compare ‘white’ and minority

ethnic respondents. • If I sample randomly I will probably get only about 10 minority ethnic

respondents. This won’t give me much of a basis for a comparison. • So I stratify my sample and sample 50/1000 minority ethnic students,

giving a probability of selection of .05• ...and 50/9,000 ‘white’ students, giving a probability of .0056• We now have 50 ‘white’ and 50 minority ethnic respondents – this is

useful because it provides more balanced information about the two sub-populations.

• However, it now looks from the sample as if the population is 50% minority ethnic, which is not the case.

• To ‘re-weight’ the responses to make them represent the composition of the ‘real’ population I can multiply each minority ethnic respondent by the inverse of their chance of selection (1000/50 = 20) and each ‘white’ respondent by the inverse of their chance (9000/50 = 180).

• These weights give a sample size that is 100 times too large (10,000/100), so dividing by 100 gives final weights of 0.2 and 1.8.