SLWK – 609 Research Methods- online- M. Secret

Podcast 3: More about Measurement

http://www.screencast.com/t/FkGKEd4Hbh

Problem Idea Theory

What How Who (variables) (methods) (sample)

Data Collection

Data Processing

Data Analysis

Discussion and Application

Backing- up.. Where are we in the research process

Measurement of VARIABLES

Problem <<<>>> Idea <<< >>> Theory What <<<>>> How <<<>>> Who (variables) (methods) (sample)

WHAT ARE YOU GOING TO OBSERVE – or measure -- some characteristic, or some type of knowledge, attitude, or behavior..

conceptualization and operationalization of variables.. How will you measure them…

A. WHAT IS MEASURMENT◦assignment of numbers to in dicators of concepts , or variables, according to rules (mathematical operations of numbers)

◦the link or the -- bridge between the world of concepts and the empirical world Tells you -- quantifies -- how much the variable

varies.. you are going from the very general or theoretical level to the fairly specific or operational level.. .science is deductive....

◦the number tells you, helps you to infer, how much or how little the concept is present

Thinking about the meaning of measurement

A variable is a concept that can be measured:

i.e. Gender is a concept.. When you measure it, it becomes a variable

Depression is a concept… when you measure it, it becomes a variable

Remember:

The variable age can be measured by answer to the question -- How old are you? ..

It can also be measured by collecting data from an existing form that gives you the date of birth

By nature, age varies by numbers of year… so the number is assigned automatically

Examples of how some variables are measured

The variable gender can be measured by the answer to the question, Are you male or female?

Because there is no automatic number associated with the variable gender, you the researcher assigns the number.. An example would be that the value of 1 is assigned to female and 0 is assigned to male

A researcher might also be able to ‘observe’ whether someone is male or female, but we all know that this would not be a particularly valid way to measure gender

The variable stress can be measured by the summed score of the answers to 10 items on the Stress scale

Stress can also be measured by some physiological measure, i.e. blood pressure, where the researcher states that a blood pressure number above a certain level is an indicator of stress

The level of social support can measured by the answer to the question ‘how much social support do you have’ 1 = very much; 2 = some; 3 = very little

Or by asking several questions about social support and adding the answers to these questions together to get one score.. This produces a scaled score, or number, that tells you how much of or how little social support exists.

Some concepts are easier to measure than other concepts because they are, by nature, more concrete

Age is fairly easy to measure by asking just one question --how old are you? Or What is your date of birth?

Depression is more difficult to measure because it’s hard to know what to count, or what exactly to observe, to know whether or not it is present.

Trauma is difficult to measure because you need to be clear about whether you’re going to measure the traumatic event or the trauma symptoms.

For abstract concepts, it is especially important that you define what you mean by a concept before you try to measure it.

i.e. Depression is a feeling of malaise or emotional discomfort that interferes with day to day routine functioning.

Trauma is an external event that causes injury or harm.. OR

Trauma is the condition or response to an event that causes injury or harm.. OR

Trauma is an emotional wound caused by…

Importance of conceptualization

What is the difference between conceptualization and operationalization?

conceptualization ◦Uses words to define◦Dictionary definition◦More abstract

Operationalization◦Uses numbers to define◦How measured◦More concrete – what you count

Measurement and operationalization can be thought of as almost the same thing

When you operationalize a concept you are assigning numbers to a variable

Operationalization = how you are going to measure a concept.

Variation is represented by the characteristics or categories or values of a particular variable

Gender is the variable, the values are male (0) or female (1);

Age is the variable, the values are 1, 2, 3,

Stess is a variable, the values are very much (3), some (2), little (3)

How a variable varies is known as the level of measurement.

Types =

Nominal – variable varies by group or categories◦(Male, female); (married, never married, divorced);

(republican, democrat)◦ variable is either present or not ..

Ordinal – varies by rank or order .. ◦More or less, higher or lower ..( Low, mid, high income

level)

Interval/ratio – varies by specific distance between values ..

◦1 is the same distance from 2 as 2 is from 3, etc etc ◦50 feet is higher than 30 feet -- ◦Difference between values/categories is precise

Levels of measurement

Example: The Group and Individual Assessment Form to “measure” individual and group participation for the research project

consider the form as a measurement tool

INDIVIDUAL ASSESSMENT OF OWN PARTICIPATION: The following scale evaluates level of effort you contributed to your Research activities. Circle the number above the statement that best describes your effort towards the project. Your name _____________________________

5 4 3 2 1 Regular attendance at meetings Attended all meetings, stayed to agreed end, working within timescale, active and attentive,

Missed many meetings, always or often late, left early, digressed, giggled, day-dreamed or gossiped most of the time

5 4 3 2 1 Contribution of ideas for the task Thought about the topic in advance of the meeting, provided workable ideas which were taken up by the group, built on others' suggestions, and were prepared to test out ideas on the group rather than keep quiet

Didn't come prepared. Didn't contribute any ideas. Tended to reject others' ideas rather than build on them

5 4 3 2 1 Researching, analyzing and preparing material for the task Did what you said you would brought materials, did an equal share of the research and helped to analyze and evaluate the material

Did no research. Didn't do what you promised to do. Didn't manage your workload. Didn't get involved with the task and allowed others to provide all the material

5 4 3 2 1 Contribution to the cooperative group process Left personal differences outside the group, willing to review group progress and tackle conflict in the group, took on different roles as needed, kept group on track, willing and flexible but focused on the task

Did not take initiative, waited to be told what to do. Always took the same role (leader, joker etc) regardless of circumstances, created conflict, and not prepared to review group processes

5 4 3 2 1 Supporting and encouraging group members Keen to listen to others, encouraged participation, enabled a collaborative learning environment, sensitive to issues affecting group members, supported group members with special needs

Sought only to complete the task, spoke over others and ignored their opinions, kept ideas and resources to self. Insensitive to individuals' needs and did not contribute to the learning process

5 4 3 2 1 Practical contribution to end product Willing to try new things. Not hogging the tasks, made a high level of contribution, took own initiative, was reliable and produced a high standard work/presentation

Not willing to take on any task, did not take any responsibilities, were unreliable so others felt the need to keep checking up, and made a limited, poor quality contribution

May want to stop the recording and access a copy of this form which is posted on blackboard

Team Assessment Form Measurement application

◦ Develop conceptual/nominal definition(s) for what this form is measuring

◦ Identify the variable(s) and the indicator(s) that the form is measuring

◦ What is the level of measurement for the variables “Regular attendance at meetings” and for “Practical contribution to end product”

◦ What are the values for each of these variables◦ How could you convert these variables to a

nominal level of measurement. ◦ Does it make sense to consider the questions on

this form as a scale. Provide a rationale for your answer.

◦ Create a scale using the items on this form ◦ Applying validity and reliability

Team Assessment Form: What is this form measuring??

In general conceptualization … Participation in group project Own/other contributions to group project

More specific conceptualization… Regular attendance at meetings Contribution of ideas for the task Researching, analyzing and preparing

material for the task Contribution to the cooperative group

process Supporting and encouraging group

members Practical contribution to end product

Regular attendance at meetingsAttended all meetings, stayed to agreed end, working within timescale, active and attentive,

Contribution of ideas for the taskThought about the topic in advance of the meeting, provided workable ideas which were taken up by the group, built on others' suggestions, and were prepared to test out ideas on the group rather than keep quiet

Researching, analyzing and preparing material for the taskDid what you said you would brought materials, did an equal share of the research and helped to analyze and evaluate the material

Contribution to the cooperative group process Left personal differences outside the group, willing to review group progress and tackle conflict in the group, took on different roles as needed, kept group on track, willing and flexible but focused on the task

Supporting and encouraging group membersKeen to listen to others, encouraged participation, enabled a collaborative learning environment, sensitive to issues affecting group members, supported group members with special needs

Practical contribution to end productWilling to try new things. Not hogging the tasks, made a high level of contribution, took own initiative, was reliable and produced a high standard work/presentation

Defining concepts.. More specific, concrete

the definition, better the measurement validity

Thought for the day!!Definition of any one concept rests on shared

understandings of other terms used in the definition

Research is as much about communication as it about anything else

Regular attendance at meetings Extent to which self/others attended all meetings, stayed to agreed end, working within timescale, active and attentive,

Contribution of ideas for the task

How much self/others thought about the topic in advance of the meeting, provided workable ideas which were taken up by the group, built on others' suggestions, and were prepared to test out ideas on the group rather than keep quiet

Researching, analyzing and preparing material for

the task How much did self/others do what he/they said; brought materials, did an equal share of the research and helped to analyze and evaluate the material

Identify the variable(s) and the indicator(s) that the form is measuringVariable

indicator

Variable

Variable

indicator

indicator

5 4 3 2 1

Regular attendance at meetings Attended all meetings, stayed to agreed end, working within timescale, active and attentive,

Missed many meetings, always or often late, left early, digressed, giggled, day-dreamed or gossiped most of the time

5 4 3 2 1 Contribution of ideas for the task Thought about the topic in advance of the meeting, provided workable ideas which were taken up by the group, built on others' suggestions, and were prepared to test out ideas on the group rather than keep quiet

Didn't come prepared. Didn't contribute any ideas. Tended to reject others' ideas rather than build on them

What are the values for each of these variables

5,4,3,2,1, = valuesHow.. or the degree to which

the variable varies

What is the level of measurement for the variables “Regular attendance at meetings” and for “Practical contribution to end product”

5 4 3 2 1

Regular attendance at meetings Attended all meetings, stayed to agreed end, working within timescale, active and attentive,

Missed many meetings, always or often late, left early, digressed, giggled, day-dreamed or gossiped most of the time

5 4 3 2 1 Contribution of ideas for the task Thought about the topic in advance of the meeting, provided workable ideas which were taken up by the group, built on others' suggestions, and were prepared to test out ideas on the group rather than keep quiet

Didn't come prepared. Didn't contribute any ideas. Tended to reject others' ideas rather than build on them

Ordinal5 is more than 4… is more than 3

Values are ordered/ranked

◦How would you create a scale using the items on this form

Add together the values for all the items into ONE score.. The one score tells you how much self/others contributed to the group effort

Application to reliability and validity In the area of measurement, researchers confront

two major issues: validity and reliability. Differentiate between validity and reliability.

Validity – does the variable measure what it is supposed to … is the operational definition the same as the conceptual definition

Reliability – do you get the same answer every time you measure a variable

What does ‘(measurement) reliability’ mean in terms of this form?

A person will answer the same way every time they answer the question -- under the same circumstances

Stable and dynamic traits◦Gender = stable.. Doesn’t change much over time

◦Happiness = dynamic… changes frequently

Reliability and Validity

What does ‘(measurement) validity’ mean in terms of this form?The form is an accurate measure of what you want it to measure….

i.e. it is a good measure of participation in/ contribution to the group effort

Test-retest◦ Will someone fill out the form on two different

days (weeks) the same way – ◦ providing nothing has changed in the situation

Inter-rater Internal Consistency reliability – applies mainly to scaled measurements (alpha or chronbach’s)

◦ Will someone answer the same way (i.e. all 5’s or all 1’s) on each item of the scale

◦ The stronger the association among individual items and the more items included, the higher the reliability of a measure

What are the ways that you could assess the reliability of this form?

Content validity/face validity subjective judgment as to whether or not an instrument is valid based

on whether or not the test items appear to be related. Content validity can be strengthe ned by having experts make the

judgment as to whether or not a measurement measures what it is supposed to measure

What are the ways that you could assess the validity of this form?

Criterion validity◦ How well does your measure compare to another measure .. Another

standard -- or ruler that you trust to be accurate . hold up your test to see how well it compares to some predetermined, already existing standard.

◦ In concurrent validity, the standard, or ruler, is other tests which have already been determined to have validity...

DEPRESSION SCALES...GCS SCALE WITH BECK OR ZUNG....or with DSMIII-R

◦ In predictive validity, the standard you are comparing your test to is what happens in the future. Do social workers who test high on the empowerment scale more likely to

become involved in community activities SAT test as a measure to determine how well students do academically and

thus compares it with how well the student actually does in school... scales to measure high or low risk adoptions placements would be compared

with whether or not the adoption was disrupted. Construct validity

◦ related to a variety of other measures as specified in a theory, used when no clear criterion exists for validation purposes

Do a cheat sheet on this.

Conceptualization and Operationalization

◦ In order to select which concepts to consider and which variables to measure, we must

Examine relevant theories Review previous research Consider the constraints and opportunities Understand that the definition of any one

concept rests on shared understandings of other terms used in the definition

Key Point Re Measurement

FROM LIT REVIEW

• What is treatment fidelity?• Why is it important?

◦ treatment is the independent variable in intervention studies.

◦ need to know the extent to which intervention was implemented in order to understand the outcome of the intervention

• What will you measure to assess treatment fidelity?

• How will you measure treatment fidelity?• What is the connection between treatment

fidelity and evidence based practice.

A Note about Treatment Fidelity

Stay tuned for the next Podcast which will be on sampling.