Quantitative Methods in Social Research 2010/11

Week 3 (morning) session28th January 2011

Data Sources: Secondary Analysis, Official Statistics &

Statistics without Surveys  

What is ‘secondary analysis’?

• Hakim: “…any further analysis of an existing dataset which presents interpretations, conclusions or knowledge additional to, or different from, those presented in the first report on the inquiry as a whole and its main results”.

• Dale et al.: “secondary analysis implies a re-working of data already analysed”.

• Hyman: “the extraction of knowledge on topics other than those which were the focus of the original surveys”.

Online course extracts: Dale et al. 1988; Dale et al. 2008

Sources for secondary analyses

• Surveys

• The Census

• Administrative and/or public records

• Longitudinal studies

• Qualitative studies

The UK Data Archive (http://www.data-archive.ac.uk) now catalogues data from surveys and qualitative studies, as well as the Census, historical data, international country-level databases, etc.

Some sources specifically geared towards secondary analysis

• The British Social Attitudes Survey• Understanding Society

http://www.understandingsociety.org.uk/• The Timescapes qualitative longitudinal study:

http://www.timescapes.leeds.ac.uk/

Benefits of secondary analysis

• It avoids costs in money and time that would make primary research impractical, especially for a lone researcher.

• It allows one to benefit from the fieldwork expertise of professional organizations.

• Cross-national and historical research become more of a practical possibility.

• Secondary analyses of longitudinal data facilitate studies of change over time.

• Large, nationally representative samples facilitate sophisticated, generalisable analyses, and (sometimes) analyses relating to small/relatively inaccessible minorities.

Bridging the quantitative/qualitative ‘divide’

• Dale et al. comment that “qualitative research can greatly enhance the value of secondary analysis by providing greater depth of information, particularly by suggesting the underlying processes that are responsible for the observed relationships”.

Why doesn’t sociology in the UK generate more secondary analyses?

• A lack of quantitatively-orientated researchers.• The legacy of critiques of quantitative research

methods.• More specifically, the legacy of critiques of

official statistics.• Although it’s now more of a question of inertia

than of ongoing scepticism?

Themes within critiques of official statistics

• Concerns about coverage

• Concerns about measurement

• Epistemological concerns

• ‘Political concerns’

A ‘damning’ quote?

“It’s [i.e. the state’s] economic and political functions are embedded in the production of official statistics, structuring both what data are produced and how this is done... only by understanding that statistics are produced as part of the administration and control of a society organised around exploitative class relations can we grasp their full meaning”

(Miles and Irvine, 1979).

However…• Analyses of official data can produce substantively

interesting results

• The producers and users of official statistics are normally very concerned about the errors in data and the data’s limitations,

• The conceptual issues arising from the use of official statistics are not dissimilar to those arising in other forms of sociological research.

• Analyses of official data have been used to critique governments with respect to issues such as unemployment, health inequalities, etc.

(The first three of the above bullet points are suggestions by Bulmer)

... nevertheless

As Hindess commented:• “Official statistics are never mere givens to be

taken as they are or else dismissed as inadequate. Like other productions they must be explained in terms of the conditions and instruments of their production”.

• “As structured social products they [i.e. official statistics] can [and should be!] be critically assessed”.

Official statistics or official data?

• Published official statistics have justifiably been viewed with some scepticism.

• However, the analysis of official data by a secondary analyst can avoid some of the problems.

• Given access to the ‘raw’ official data, she or he can manipulate them in ways different to how they were processed to produce published official statistics.

Are UK official statistics getting more independent?

• A Statistics Board resulting from the Statistics Bill of July 2007, renamed the UK Statistics Authority in February 2008 (see: http://www.statisticsauthority.gov.uk/) is:

• “... an independent body operating at arm's length from government as a non-ministerial department, directly accountable to Parliament.  … [its] overall objective is to promote and safeguard the quality of official statistics that serve the public good. It is also required to safeguard the comprehensiveness of official statistics”.

Some more specific developments: I

• OPCS [now ONS] Disability Surveys were criticised for not adequately reflecting disabled people’s perspectives on their disabilities.

(see Abberley’s chapter in Levitas and Guy, 1996).

• However, they were nevertheless used for some interesting and useful secondary analyes (see Pole and Lampard, 2002, Ch. 7).

• More recently, the Office for Disability Issues (ODI) brought together a group of disabled people as a reference network, in part to facilitate the effective design of a new longitudinal disability survey, the Life Opportunities Survey (LOS):

(see http://www.ons.gov.uk/about/surveys/a-z-of-surveys/life-opportunities-survey/index.html)

Some more specific developments: II

• A number of UK government surveys now (since 2009) ask a question on sexual orientation, following a question being asked in the 2007 Citizenship Survey (and resulting from ONS's Sexual Identity Project, established in 2006).

• This development reflects more general governmental concerns about the availability of ‘equality data’.

• However, it does not seem that a question will be asked on this topic in the 2001 Census!

• The consensus also seems to be that the results generated by the question will under-estimate non-heterosexual orientations.

“What is the moral? Must have a moral…”

• Whether the source of their data is official or non-official, secondary analysts should gain an extensive knowledge of the research design and data collection process.

• This allows the secondary analyst to adopt an informed and suitably critical approach to their assessment of the validity and value of their data source(s).

The data source for the slide title (I think) is “A Funny Thing Happened on the Way to the Forum” (Sondheim)

Key issues in secondary analysis(according to Dale et al.)

• What was the original purpose of the study and what conceptual framework was used? Who was responsible for collecting the data?

• What data did the study collect and how were variables such as occupational class operationalized?

• What was the sample design that was used and what was the level and pattern of non-response?

Is there documentation available in relation to?

• Sample selection

• Patterns of (non-)response

• Interview schedules [Questionnaires]

• Instructions to interviewers

• The coding of answers

• The construction of derived variables

Some other relevant questions...• Is secondary analysis an appropriate approach given the

researcher’s objectives?

• Does the secondary analyst know the topic area well enough to be able to interpret and evaluate the information available?

• What similarities and differences are there between the conceptual frameworks of the original researchers and of the secondary analyst?

• Are the data recent and extensive enough for the secondary analyst’s purposes?

• How consistent is the information with information from other sources?

• Is the information representativeness enough to support generalisations? Is weighting needed to correct for a lack of representativeness?

An example: the General Household Survey

Advantages of the GHS include• a large sample size• the fact that it has been repeated more or less

annually since the early 1970s, which allows trends to be examined

• a broad agenda which means that relationships between concepts belonging to different policy areas can be examined

• a hierarchical structure, which allows linkages between different members of the same household to be examined (Dale et al.)

Some examples of sources and issues from Richard’s research

• Social Change and Economic Life Initiative (SCELI) main survey (1986)

• National Survey of Sexual Attitudes and Lifestyles II (2000) [and various other couple-related surveys]

• General Household Survey (1991 & 2005)• British Election Study (1987)

See also Pole and Lampard, 2002, Ch. 7.

Reasons for the end of a cohabiting or marital relationship

(as shown on a NATSAL II showcard)• Unfaithfulness or adultery

• Money problems

• Difficulties with our sex life

• Different interests, nothing in common

• Grew apart

• Not having children

• Lack of respect or appreciation

• Domestic violence

• Arguments

• Not sharing household chores enough

• One of us moved because of a change in circumstances

(for example, changed jobs)

• Death of partner

• Another reason (please say what)

...and the categories that had to be added

• Drink, drugs or gambling problem• Mental health or related problem• Problem with children/step-children• Never at home (e.g. always out with friends)• Problems with parents/in-laws/family• Age-related problems (e.g. big age difference)• Another relationship involved• Lived in/moved to a different country/area• Still in relationship, but stopped living together• Change of mind/feelings/personality• Partner just left without any explanation

Statistics without surveys• There are methods of quantitative analysis that do not rely on

surveys. Three that we will discuss are:– Content analysis (to which Eric will return in Week 4)

– Analysis of comparative-historical materials– Observation(al) studies

• All involve the operationalisation of concepts and coding of data, as well as decisions about sampling and so none are immune from criticisms aimed at these processes, and the subjectivity involved therein.

• But since all three largely involve unobtrusive methods, they tend not to involve the (artificial, potentially power-laden, and much criticised) interactions found in survey interviews.

• We will conclude by looking briefly at network analysis. This is actually a particular method of statistical analysis, but one that has been developed in relative isolation to mainstream statistics, and one that has different starting assumptions and utilizes different sorts of data sets.

Data CollectionQualitative vs. Quantitative Methods

Qualitative Quantitative

Observing Participant observation Structured observation

Talking to peopleIn-depth interviews

& Focus groupsSurveys

Looking at ‘texts’ (books, films, web pages,

adverts…)Discourse analysis Content analysis

Using existing information

Comparative -historical research

Analysis of existing statistics/data

OtherExperiments / quasi-

experiments (not common)

Note: Network Analysis is largely quantitative, but involves a whole set of different analytic techniques – data can be from surveys, structured obs, or content analysis.

Content Analysis

Method of transforming symbolic content of a document (such as words or images) from a qualitative unsystematic form into a quantitative systematic form.

See Bryman, 2008, Ch. 12 (online course extract)

Possible Units of Analysis for Content Analysis

…but a unit of analysis may also be:a film, a scene, a TV episode, a wall (containing graffiti),a rubbish bin, a politician’s speech, a web-site,or a blog posting…

Comparative-Historical Research• Much comparative-historical research does not use statistics. • However if you are looking at change over time or are comparing

different countries or regions there are a large number of statistics that can be used:

• Macro-level secondary statistics – e.g. World Bank “development indicators” i.e. mortalitity rates; televisions per 1000 population; Literacy rates. Or “OECD Main Economic Indicators” – i.e. foreign direct investment; GDP; GNP… etc.[See the Library Statistics Workbook that is linked to the module web page; this is of value both in terms of accessing international data and statistical sources generally]

• Primary statistics – these are datasets that you construct for yourself from historical and comparative research. They may document anything from the strength and political composition of particular trade unions in a particular time and place; to land-holding patterns in different regions as described by local tax-records; to speeches made by Vice-Chancellors of UK universities at public forums over the last century… To conduct quantitative analysis of primary historical research it just needs to be systematically coded.

Sampling Comparative-Historical Events

• If you are going to use comparative-historical data to create a dataset it is important to think about whether you have data from the entire population of events that you are interested in (i.e. every strike that occurred in the UK between 1990 and 2000), or whether you are focusing on a subset (thirty strikes that occurred in the UK between 1990 and 2000).

• If you present statistical information for a subset of events you are sampling and the same issues of occur as any other time that you sample data: your findings are only statistically generalisable if the sampling is random (or if each event has a known - typically equal - probability of selection into the subset).

• On the other hand, there are often substantive reasons to choose specific “important” events to be part of your subset (i.e. large-scale strikes that involved media campaigns). This is legitimate and statistics gleaned from these may be interesting and informative. However they are not statistically generalisable to all events (i.e. strikes generally) and so inferential statistics are not appropriate.

Observation(al) studies• Observation is not just the preserve of qualitative

methods. Quantitative methods can be applied where structured or systematic observation is carried out.

• Like qualitative observation studies (and surveys), this involves cross-sectional data (we can only observe the present).

• Unlike qualitative observation, structured or systematic observation is not inductive but requires the prior determination of what to observe (although this may be suggested by initial unstructured observations).

See Pole and Lampard, 2002, Ch. 4.

The observation schedule• To produce quantitative data an observation schedule or coding

scheme is required.

• This describes what is to be observed and how what is observed should be coded.

• For example, if I were observing in the Library Café and was interested in interactions between students and the staff working at the cash-registers I could code each student’s behaviour in the following way:

1. No conversation, no eye contact, no smile

2. Eye contact and/or smile, no conversation

3. Conversation, only as required by the transaction

4. Conversation as required by the transaction and polite thanks.

5. Conversation that goes beyond transaction and polite thanks.

• The observations must be focused – and relevant to the research question

• The schedule (like closed questions in a questionnaire) should have categories that are mutually exclusive and exhaustive

• Recording should involve as little observer interpretation as possible – this is where reliability is diminished.

The observation schedule

Sampling in Structured Observations

• It is important to be clear about the unit of analysis – are you sampling events/situations, interactions, or individuals?

• Sampling must consider the dimension of time in determining who, where, and when to make observations. It may sometimes be appropriate to sample at multiple time periods and in multiple sites.

Benefits and Drawbacks of Structured Observation I

• Like other ‘unobtrusive measures’ structured observation may avoid researcher contamination – enabling the study of people in their natural environment.

• Unlike surveys it does not depend on the negotiation of meaning between interviewer and interviewee (or the interviewee’s accurate representation of her behaviour).

• Unlike qualitative observation studies it can produce relatively reliable data and since observation (with a schedule) can be undertaken by more than one researcher, it enables large-scale data collection.

Benefits and Drawbacks of Structured Observation II

• However the researcher will only ‘see’ the predetermined categories of action that the schedule specifies. These may not be the categories of action that are relevant to participants.

• Since structured observation precludes questioning participants about their motives or opinions, it is wholly dependent on observing behaviour and on the ability of the researcher to appropriately assess this.

• It is ahistorical, in that it can only assess behaviour in the moment (unlike surveys which can ask, albeit imperfectly, about people’s pasts, or other methods such as content analysis, historical or secondary data analysis).

Network Analysis

• Network Analysis is based on the assumption that people’s actions are interdependent and so it is critical to describe the networks of relationships that exist.

• It is characterized by a distinctive methodology encompassing techniques for collecting data, statistical analysis, visual representation, etc

• Critically, network analysis uses Matrices to analyse the relationships between people, organisations and institutions. It also uses graph theory.

Network analysis is concerned with attributes of pairs of individuals, of which binary relations are the main (but not only kind.

Some examples of dyadic attributes:

• Kinship: brother of, father of • Social Roles: boss of, teacher of, friend of • Affective: likes, respects, hates • Cognitive: knows, views as similar • Actions: talks to, has lunch with, attacks • Flows: number of cars moving between • Distance: number of miles between • Co-occurrence: is in the same club as, has the

same colour hair as

The relationships needn’t be between individuals

Ties could be…

• between corporations, or

• between political organizations, or

• between community groups, or

• any combination of these.

Divided We Stand Political books were selected from the New York Times Bestseller List as starting points for 'snowball sampling'. Two books are linked in the network if they were purchased by the same person -- "Customers who bought this book also bought:". The pattern reveals two distinct clusters with dense internal ties. (early 2004)

Are these two clusters connected by non-political books? In the map there is a path of 4 steps from the most central Blue book to the most central Red book. Using current fiction titles we do not find a shorter path! Using Da Vinci Code the centers of the clusters are 7 degrees/steps apart, The Five People You Meet in Heaven and South Beach Diet result is 9 degrees apart and The Last Juror takes over 15 steps to connect the centers.

Weaknesses in Network Analysis

• It is difficult to get information on complete networks (as this involves getting information from all individuals/organizations). This is required for many of the methods involved.

• Network analysis has been criticised for being better at analysing relationships between people (or ‘nodes’) than the structural and material aspects of power.

Network analysis games…• Network analysis has been used to develop six

degrees of Kevin Bacon (the parlour game developed from the notion that everyone is separated from everyone else by just six degrees of separation). The aim in Six Degrees of Kevin Bacon is to link any movie star to Kevin Bacon via films that they have both been in in less than six steps.

• Can you think of anyone who is more than three degrees of separation from Kevin Bacon?

• You can check your answer at: http://oracleofbacon.org/. This site also allows you to link any other stars together.