chapter five: nonexperimental methods ii: ex post facto studies, surveys and questionnaires,...

Chapter Five: Nonexperimental Methods

II: Ex Post Facto Studies,

Surveys and Questionnaires,

Sampling and

Basic Research Strategies

Ex Post Facto Studies

Ex post facto – is a Latin phrase meaning “after the fact.”


Ex post facto – is a Latin phrase meaning “after the fact.”

When we conduct an ex post facto study, we are using an IV “after the fact” – it has already varied before we arrived on the scene.


Surveys, Questionnaires, Tests, and Inventories

Surveys, Questionnaires, Tests, and Inventories Surveys typically request our opinion on some topic or

issue that is of interest to the researcher.

Surveys, Questionnaires, Tests, and Inventories Types of Surveys


Descriptive


Descriptive – seeks to determine what percentage of the population has a certain characteristic, holds a certain opinion, or engages in a particular behavior.



Examples:



Examples: The Gallup Poll



Examples: The Gallup Poll Nielsen television ratings


Descriptive Analytic


Descriptive Analytic –seeks to determine what the relevant

variables are and how they might be related.



variables are and how they might be related. Questions need to be chosen carefully.



variables are and how they might be related. Questions need to be chosen carefully. Pilot testing is generally necessary to determine the

type of questions that will be used in the final survey instrument.

Surveys, Questionnaires, Tests, and Inventories Developing a Good Survey or Questionnaire (see table

5-1, p. 89).

Surveys, Questionnaires, Tests, and Inventories Step 1

Surveys, Questionnaires, Tests, and Inventories Step 1 – decide what type of instrument to use. How

will the information be gathered?


will the information be gathered? Will you use a mail survey, conduct a phone interview, or

conduct the survey in person? Will you use trained interviewers?

Surveys, Questionnaires, Tests, and Inventories

Step 1 – decide what type of instrument to use. How will the information be gathered?


will the information be gathered? Step 2 – identify the types of questions to use.



Yes-No Questions



Yes-No Questions The respondent answers yes or no to the items.




Forced Alternative Questions




Forced Alternative Questions The respondent must select between two alternative

responses.





responses. Multiple-Choice Questions






The respondent must select the most suitable response from among several alternatives.







Likert-Type Scales

Surveys, Questionnaires, Tests, and Inventories Step 2 – identify the types of questions to use.





Likert-Type Scales The individual answers a question by selecting a

response alternative from a designated scale. A typical scale might be the following: (5) strongly agree, (4) agree, (3) undecided, (2) disagree, or (1) strongly disagree.

Surveys, Questionnaires, Tests, and Inventories Step 2 – identify the types of questions to use.




Likert-Type Scales The individual answers a question by selecting a

response alternative from a designated scale. A typical scale might be the following: (5) strongly agree, (4) agree, (3) undecided, (2) disagree, or (1) strongly disagree.

Open-Ended Questions

Surveys, Questionnaires, Tests, and InventoriesStep 2 – identify the types of questions to use.

Multiple-Choice Questions The respondent must select the most suitable response

from among several alternatives. Likert-Type Scales

The individual answers a question by selecting a response alternative from a designated scale. A typical scale might be the following: (5) strongly agree, (4) agree, (3) undecided, (2) disagree, or (1) strongly disagree.

Open-Ended Questions A question is asked to which the respondent must

construct his or her own answer.


will the information be gathered? Step 2 – identify the types of questions to use. Step 3


will the information be gathered? Step 2 – identify the types of questions to use. Step 3 – write the items: They should be clear, short,

and specific.

Surveys, Questionnaires, Tests, and Inventories Step 3 – write the items: They should be clear, short,

and specific. Use familiar vocabulary and be at the reading level of the

individuals you intend to test.



and specific. Step 4



and specific. Step 4 – Pilot-test and seek opinions from

knowledgeable others.

Surveys, Questionnaires, Tests, and Inventories Step 4 – Pilot-test and seek opinions from

knowledgeable others. It is important to ask others, especially professionals who

have expertise in your area of research interest, to review your items. They may be able to detect biases and unintended wordings that you had not considered.




knowledgeable others. Step 5




knowledgeable others. Step 5 – determine the relevant demographic data to

be collected.

Surveys, Questionnaires, Tests, and Inventories Step 5 – determine the relevant demographic data to

be collected. Includes items such as age, sex, annual income, size of

community, academic major, and academic classification.





be collected. Step 6





be collected. Step 6 – determine administration procedures and

develop instructions.

Surveys, Questionnaires, Tests, and Inventories Step 6 – determine administration procedures and

develop instructions. Instructions must be clear, concise, easy to follow, and

thoroughly rehearsed (for in-person or phone interviews). How will informed consent be obtained?

Surveys, Questionnaires, Tests, and Inventories Survey Administration Procedures


Mail Surveys


Mail Surveys Advantages



The researcher does not have to be present while the survey is being completed.



The researcher does not have to be present while the survey is being completed.

Surveys can be sent to a much larger number of participants than a single researcher could ever hope to contact in person.


Mail Surveys Disadvantages



The researcher cannot be sure who actually completes the survey.




There is no guarantee the respondent answered the questions in the same order in which they appeared in the survey.




There is no guarantee the respondent answered the questions in the same order in which they appeared in the survey.

Low return rate.

Surveys, Questionnaires, Tests, and Inventories Personal Interviews


Advantages When a trained interviewer administers a survey in a

respondent’s home, the response rate climbs dramatically.


Advantages When a trained interviewer administers a survey in a

respondent’s home, the response rate climbs dramatically.

The trained interviewer is able to clarify ambiguous questions, making sure that all questions are answered in the proper sequence, and generally assisting with any problems that the respondents may experience.


Disadvantages


Disadvantages The potential for considerable expenditure of time and

money exists.


Disadvantages The potential for considerable expenditure of time and

money exists. People not always at home.

Surveys, Questionnaires, Tests, and Inventories Telephone Interviews


Advantages


Advantages It is easy to create random samples with random-digit

dialing.



dialing. 95% of households in the U.S. have telephones.



dialing. 95% of households in the U.S. have telephones. Computer technology allows responses to be entered

directly as they are made.


Disadvantages


Disadvantages Incoming calls may be screened and potential

respondents lost.



respondents lost. It is easy to say no to a phone interviewer.



respondents lost. It is easy to say no to a phone interviewer. Cannot use visual aids to clarify questions.



respondents lost. It is easy to say no to a phone interviewer. Cannot use visual aids to clarify questions. Cannot evaluate nonverbal cues such as facial

expressions, gestures, and posture.



respondents lost. It is easy to say no to a phone interviewer. Cannot use visual aids to clarify questions. Cannot evaluate nonverbal cues such as facial

expressions, gestures, and posture. It is more difficult to establish rapport.

Surveys, Questionnaires, Tests, and Inventories Tests and Inventories

Surveys, Questionnaires, Tests, and Inventories Tests and Inventories

Tests and inventories are designed to assess a specific attribute, ability, or characteristic possessed by the individual being tested.

Surveys, Questionnaires, Tests, and Inventories Characteristics of Good Tests and Inventories


Validity


Validity – a test or inventory has validity when it actually measures what it is supposed to measure.


Validity (ways to establish) Content validity


Validity (ways to establish) Content validity – indicates that the test items actually

represent the type of material they are supposed to test.


Validity (ways to establish) Content validity – indicates that the test items

actually represent the type of material they are supposed to test. A panel of expert judges is often used to assess the content validity of test items.


Validity (ways to establish) Content validity Concurrent validity


Validity (ways to establish) Content validity Concurrent validity – is established when a score on a

test or inventory agrees with another measure (e.g. aggression score on a test agrees with a clinician’s assessment).


Validity (ways to establish) Content validity Concurrent validity Criterion validity


Validity (ways to establish) Content validity Concurrent validity Criterion validity – is established when a test score

compares favorably with a predicted future outcome (e.g. SAT score and college success).

Surveys, Questionnaires, Tests, and Inventories Reliability


Refers to the extent that the test or inventory is consistent in its evaluation of the same individuals over repeated administrations.


Refers to the extent that the test or inventory is consistent in its evaluation of the same individuals over repeated administrations.

The greater the similarity between scores produced by the same individuals on repeated administrations, the greater the reliability of the test or inventory.

Surveys, Questionnaires, Tests, and Inventories Reliability (assessment)


Test-retest procedure


Test-retest procedure The test is simply given a second time and scores from

the two tests are compared; the greater the similarity, the higher the reliability


Test-retest procedure Split-Half technique


Test-retest procedure Split-Half technique

Involves dividing a test or inventory into two halves or subtests and then administering them to the same individuals on different occasions or by administering the entire test and then splitting it into two halves.

Surveys, Questionnaires, Tests, and Inventories Types of Tests and Inventories


Achievement tests


Achievement tests Are given when an evaluation of an individual’s level of

mastery or competence is desired.


Achievement tests Are given when an evaluation of an individual’s level of

mastery or competence is desired. The score that distinguishes passing from failing

determines the minimum level of achievement that must be attained.


Achievement tests Aptitude tests


Achievement tests Aptitude tests

Are used to assess an individual’s ability or skill in a particular situation or job.


Achievement tests Aptitude tests Personality tests or inventories


Achievement tests Aptitude tests Personality tests or inventories

Measure specific aspects of an individual’s motivational state, interpersonal capability, or personality.

Sampling Considerations and Basic Research Strategies Sampling

Sampling Considerations and Basic Research Strategies Sampling

Deals with the question of who will participate in our research project and whether they are a representative group.

Sampling Considerations and Basic Research Strategies Research Strategies


The main strategies used by researchers are the:


The main strategies used by researchers are the: Single strata approach


The main strategies used by researchers are the: Single strata approach Cross sectional approach


The main strategies used by researchers are the: Single strata approach Cross sectional approach Longitudinal approach

Sampling Considerations and Basic Research Strategies Population

Sampling Considerations and Basic Research Strategies Population

The complete set of individuals or events that we want to represent.

Sampling Considerations and Basic Research Strategies Sample


The group that we select to represent the population is called the sample.



Random sample



Random sample Every member of the population has an equal likelihood

of being included in the sample.

Sampling Considerations and Basic Research Strategies Random sample

Every member of the population has an equal likelihood of being included in the sample.

Random sampling without replacement



Random sampling without replacement The member of population chosen for the sample is not

eligible to be selected again once returned to the population.





Random sampling with replacement Member of population chosen for sample is eligible to be

selected again once returned to the population.





Random sampling with replacement Member of population chosen for sample is eligible to be

selected again once returned to the population. Stratified random sampling




eligible to be selected again once returned to the population. Random sampling with replacement

Member of population chosen for sample is eligible to be selected again once returned to the population.

Stratified random sampling Involves dividing the population into subpopulations or strata

and then drawing a random sample from one or more of these strata.

Sampling Considerations and Basic Research Strategies Basic Research Strategies


Single-strata approach


Single-strata approach Seeks to acquire data from a single, specified segment

of the population.



of the population. Cross-sectional research




Involves the comparison of two or more groups of participants during the same, rather limited, time span.





Longitudinal research





Longitudinal research Involves acquiring data from a single group of

participants over a long period of time.

chapter five: nonexperimental methods ii: ex post facto studies, surveys and questionnaires,...

Documents

inventories step

gallup poll slide

certain opinion

certain characteristic

particular behavior

type of instrument

type of questions

relevant variables