SII: The Formation of Social Facts, or Who

Counts What?Tuesday, 27th January 2007

Outline

• Survey Research – Criticisms• Secondary Analysis and Official

Statistics• Content Analysis – a method of

producing quantitative data without surveys

• Quantitative Research – an evaluation.

Survey Research

The ‘positivist’ assumptions:1. It is possible to discover facts about people’s

actions, attitudes and attributes by asking them questions and recording their answers systematically.

2. The facts that we gather can be used to test our theories.

3. Statistical procedures can be used to unveil or clarify truths about the social world.

4. Questionnaires – the instruments for collecting facts in social surveys – are not inherently biased.

The Problems with Surveys…

• Phenomenological Critique• Marxist Critique• Feminist Critique

Survey data is socially constructed in the interaction between interviewees (trying to work out what questions mean and give appropriate

answers) and interviewers (interpreting, classifying and coding answers). It does not represent “facts.”

Phenomenological Critique of Surveys 1

Phenomenological Critique of Surveys 2

Quantification is not valid as it does not represent the way that people think about their lives (it is imposed by the researcher). i.e. we

don’t think that we’re a 3.6 or 7.3 on an “interest in politics” scale.

Note: when natural scientists construct scales and measure things it is not a problem, but a human

being is not a rock or a tree. Human beings construct first-person accounts to explain their

behaviour and sociologists should take these into consideration.

Marxist Critique of Surveys 1

Surveys classify people according to categories derived from a bourgeois world – they therefore reproduce existing ideological representations of

reality, rather than challenging them and so perpetuate the existing system of class

domination

Marxist Critique of Surveys 2

Surveys objectify people – treating them instrumentally, as units of information, not as

human beings.

Furthermore, since the capitalist state is one of the few institutions that can afford to carry out

surveys, it is able to use them to gather information that will further the oppression of the

masses.

Feminist Critique of Surveys 1

The inherently hierarchical researcher-respondent relationship has perpetuated and reinforced women’s oppression. Since historically women have not been researchers, nor devised surveys they have not been able to

“speak for themselves” but have been categorized, and put into inferior classificatory positions by male investigators, basing their

classifications on male norms and roles.

Feminist Critique of Surveys 2

Numbers are a masculine means of expression. For women to speak for themselves they require methods that emphasize empathy, intuition, and intersubjective understanding, not the reduction of people to charts, numbers and

percentages.

So… are surveys always biased, oppressive and meaningless?

• Some of the phenomenological critiques about the social construction of meaning have alerted survey writers to be more careful about questions, and to recognize the role of the interviewer. However problems of meaning and intersubjectivity remain

• Some surveys do use the language of the bourgeois world, however it is not necessarily the case that they must – and working class or other oppressed groups could conduct their own surveys (to serve their interests). On the other hand, large-scale surveys require extensive funding and this is unlikely to be at the disposal of oppressed groups.

• Categories and classifications could be better grounded in women’s and others’ experiences – therefore better allowing interviewees to speak in their own voices. This is happening to some extent – i.e. there are questions on many surveys now about hours spent doing household chores. Overcoming this may be in part a technical problem.

• It is hard to maintain that quantification is masculine without going down a very rapidly sexist and biologically determinist slope.

Nonetheless it is worth keeping these critiques in mind and using survey data critically – recognizing that sometimes it is not the most appropriate method.

Secondary AnalysisAdvantages• Cheap• Quick more time

for analysis• Good quality data

availability• Large samples

subgroup analysis• Possible to study

prior time periods • Possible to do cross-

cultural study• New topics can be

addressed or old topics reanalysed with new techniques

Disadvantages• Lack of control over

quality• Lack of familiarity

with data slow analysis

• Complexity of large-scale datasets difficulties for analysis

• Inappropriate variables/question wording, or absence of key variables

Secondary Data Analysis: 6 Questions to Ask

(from Dale et al 1988)1. What was the purpose of the study? What was the

practical rationale? Was there a guiding theoretical or conceptual framework?

2. What information has been collected? Is it what you’re interested in? Do the categories enable you to get at what you want to investigate?

3. What sampling frame was used and what’s the sampling unit (individuals, households, businesses…)? Are there potential biases? What is the response rate?

4. What are the credentials of the person, or organization that collected the data? What is the quality of the data?

5. Is the survey nationally (or internationally, or regionally) representative? Will it support generalizations?

6. When was the data collected? Is it still relevant?

Official Statistics are problematic to use in social research because they…

• Represent the interests of the state.

• Are unlikely to use categories or generalizations that sociologists are interested in, or may do this poorly.

• Reflect particular social policies – i.e. over what is a crime, or how to classify unemployment (over the 1980s and 1990s the unemployed were categorized in official statistics as those claiming benefits). This meant that the number of unemployed rose (or more often fell) as entitlement to benefits was changed).

• Are socially consequential and therefore the outcome of particular social struggles and victories – i.e. arguments about how to classify different ethnic groups reflect the fact that being ‘counted’ enables comparison and perhaps the allocation of additional resources. Therefore the inclusion of some ethnic groups and not others in classificatory schemes is not the product of an objective determination but the outcome of historically specific social struggle.

Different People’s Relationship to Official Survey Data

Government statisticianPrimary Analyst

Academic researcherSecondary Analyst

Member of PublicUser of tabulations

CONTROL(Set agenda; specify questions)

YES NO NO

KNOWLEDGE OF PROCESS(Scope for assessment and criticism)

YES YES/MAYBE NO/MAYBE

ACCESSIBILITY(Form of data available)

RAW DATA (CLOSE TO) RAW DATA

PUBLISHED FIGURES

The Use of Official Statistics: Trends in the UK

(from Levitas and Guy 1996)

• 1950s-1970s: growing scepticism among sociologists for philosophical and political reasons (measurement; objectivity; the state). Critique focusing especially on measures of crime and deviance.

• 1980 turning point: Bulmer’s article; Social statistics (official): developments 1970s, Rayner review 1980/1

• (Relationships between) relevant groups of people: Ministers / govt. statisticians / academics / public

• 1980s: political interference / secondary analysis explodes

The independence of the Office for National Statistics

Brown to give ONS independence

Matthew Tempest, political correspondentThe Guardian, Monday November 28, 2005

“The Office for National Statistics is to be made independent of government, Gordon Brown said today. The chancellor's surprise announcement came during a speech to business leaders at the annual CBI conference in London.

Mr Brown likened the move to his decision, soon after Labour's 1997 election victory, to make the Bank of England independent.

The Conservatives and the Liberal Democrats have long called for the body, which produces key data, to be freed from government control. Mr Brown's decision also reflects growing public mistrust in official government figures, which only 17% of people believe are produced without political interference, according to a recent Mori survey…”

Article continues…

On 26 July 2007, the Statistics and Registration Service Act received Royal Assent heralding a new era of independence for both the Office for National Statistics and for the UK statistical system more widely.

The Act, which comes into force on 1 April 2008, provides for the creation of a new independent body, the Statistics Board, which will operate at arm's length from Ministers as a Non-Ministerial Department accountable to Parliament. The Board has a statutory responsibility to promote and safeguard the production and publication of official statistics that serve the public good; and the quality, good practice and comprehensiveness of official statistics. The Board's responsibilities will cover the whole UK statistical system, including England, Scotland, Wales and Northern Ireland. Continues…

See Criticism in Radical Statistics

Content Analysis

Method of transforming symbolic content of a document (such as words or images) from a qualitative unsystematic form into a quantitative systematic form.

Possible Units of Analysis for Content Analysis…but a unit of analysis may also be:a film, a scene, a TV episode, a wall (containing graffiti),a rubbish bin, a politician’s speech, a web-site,or a blog posting…

Sampling in Content Analysis

You can use random sampling methods in content analysis just like you can in surveying people.

For example:

You are interested in whether politicians have become increasingly or decreasingly respectful of their opponents over the last decade. And you decide to study speeches made in Parliament.

You could study every speech made over the last ten years. This would capture the entire relevant population.

However if you did not have the resources to read every speech…

You could randomly select a month (say February) and then randomly select days in that month (say 13th, 16th, 23rd and 27th), and then study every speech given on those days (or the next weekday after these days if they fell on a weekend) in each year of the ten years in the study.

Manifest vs. Latent Content

Manifest ContentThe objective, surface, or concrete content

Examples:-Number of times the word homosexual appears in the newspaper

-Number of times someone drinks alcohol in a TV show

-Number of pictures of women in a text-book

Latent ContentThe underlying or implicit meanings

Examples:-How approvingly or disapprovingly homosexual behavior is mentioned in a newspaper

-How intoxicated someone becomes after drinking alcohol on a TV show

-Whether women are performing masculine of feminine tasks when they are pictured in a text-book

Coding in Content AnalysisCategorizing raw data (behaviors or elements) into

a limited number of standardized categories, suitable for analysis

You need to develop your own coding scheme that:

• May be based on existing coding schemes or on your orienting theories

• May emerge (inductively) from looking at the data

• Must be numerical in order to analyze it statistically (even where you are coding latent content)

• Must have categories that are:1. exhaustive2. mutually exclusive3. theoretically relevant

seeking Eric, consultant, American with French MBA - m4m Date: 2006-02-25, 6:44PM GMT/BST

We met Jan 13th at a pub. You were with two friends, who my friend impressed with his knowledge of soccer. Wanted to know you more. Will return to London and want to contact you. Please write to me.

Norwegian girl in Blue Posts, Berwick St., Wednesday night - m4w Date: 2006-02-25, 2:47PM GMT/BST

I gave you a light (and later a cigarette) and commented on your English - I still don't know if you were really Norwegian or if it was a joke, so if you're reading this, put me out of my misery! ;)

South African lady on the tube with your dad Date: 2006-02-25, 12:26AM GMT/BST

Hi! you were going to Liverpool street station to change for St Pauls and thereafter you were going to see Phantom in Lecester Squre.

I misheard you and offered you advice on how best to get to Leicester Square, missing out Liverpool Street. I would love to take you out for a meal!

Holborn station, today at around 9 am - w4m - 30 Date: 2006-02-23, 9:06PM GMT/BST

We exchanged looks while entering Holborn station, both heading to to the city, I guess. You smart and stylish professional with beautifully deep eyes, me cute, thin blonde in a black designer coat.

Reading Kundera on the 94 bus last march/april - w4m Date: 2006-02-19, 6:11PM GMT/BST

We were on the 94 bus in the evening; you were tall, young, handsome in suit reading Milan Kundera;I am mid forties curly hair, slender attractive and foreign. I was sitting at the single seat facing the wrong way and I spoke to you about the book and found out you work in the City to earn a living but really interested in literary job; you got off early on before The Bush restaurant and into a side street; been hoping to catch you again but no, email me!!

Mention when ‘connected’:1. Within last week2. Within last month 3. Within last year4. Longer than a year5. No date mentioned

“Missed Connections” From: www.london.craigslist.org Feb 19th-25th 2006

Coding – examples:

Poster describes him/herself physically:

0. No1. Yes

3

1

1

5

4

0

0

0 1

1

Sample table describing findings

Strengths of Content Analysis

• Economy of time and money.• Easy to repeat a portion of the

study if necessary.• Permits study of processes over

time.• Researcher seldom has any effect

on the subject being studied.• Reliability – consistent results over

time (especially with manifest content).

Weaknesses of Content Analysis

• Limited to the examination of recorded communications.

• Problems of validity are likely – are the sources meaningful measures of what we want to measure? – for example: is what appears in the media a good representation of political currents? Is what’s written on Warwick blogs a good representation of student opinions?

• Latent measures inevitably involve subjective interpretation.

Against Quantitative Methods

• Quantification is inherently positivist – it assumes an external and measurable reality; treating social relations as ‘things.’

• Quantification is not meaningful – it substitutes numeric values for actors’ accounts and in so doing ignores what is central to social relations.

• Quantification lends a veil of (false) scientific objectivity to the study of sociology – There’s a lot more wiggle room than the presentation of scientific ‘tests’ suggests. Neither the collection nor analysis of data is value neutral and favoured theories are rarely discarded when ‘disproved.

• Quantification produces lazy researchers – social researchers are loathe to study things that cannot (or have not) been easily quantified. Researchers are also likely to orient their research questions to the particular methods of analysis that are most often taught or easiest to do on popular statistical programs.

For Quantitative Methods• Most social science (not just quantitative methods) seeks to discover some

sort of external reality – therefore the charge of positivism-light could be made more widely than quantitative methods (and ethnography cannot really escape).

• Population characteristics do matter. And sometimes it is very useful and/or effective to know a small amount of information about a large number of cases. Quantitative studies are well suited to this - for example last year there were reports in the press that women working full-time earn an average of 17% less than men in comparable jobs and women working part time earn 31% less. While this doesn’t tell you the causal processes, it does spotlight socially relevant relationships.

• In some contexts numbers are relevant. There are lots of social relations that are quantifiable in some way or other – age; voting proportions; wages; time, etc. The investigation of social phenomena that involve numerically defined constructs will of necessity involve quantitative methods.

• The epistemological resistance of many sociologists (especially in the UK) to quantitative methods is often partially rooted in a reluctance to engage with the maths involved. To the extent that the consequences of disengagement is researchers (especially female) without an understanding of quantitative methods it reproduces their relative (inter- and intra-disciplinary) disempowerment.

• In practice many people choose research methods on the basis of practical rather than (or as well as) epistemological considerations. A critical engagement with quantitative methods expands the realm of possible choices open to the social scientist.